Dangerous turn

[Daniel Shawul]

I've sent 12.75 to TCEC for future events (I'm sure they'll remove Ethereal soon via the 'not developed' rule because the leadership dislikes me for fighting for whats right).

I shared 12.75-SFNNUE with them as well. TCEC has two options: Open the field with NNUEs and allow anyone to do it, thus allowing all engines to easily retrofit NNUE and gain similar elo. Or to bar NNUE entries that are not from Stockfish. You can't have it both ways without some hypocricy. Unless of course they want to ban the engines that have trained on Stockfish data.

I've left the TCEC Discord, and asked that they not contact me again. They are hands down the most unprofessional tournament for any game or event I have ever seen. They do not make rules to fit the needs of the competition. They make rules to fit the needs of the engines they want in the competition.

I was planning to obfuscate the Stockfish Network and send that into TCEC, absolutely demolish Leela and Allie, which would cause the directors to take some steps. I don't think I care enough to do it. I don't need to prove the shameless favoring of Leela and Allie -- its apparent to everyone.

I hope TCEC crashes and burns and something better replaces it. CCC has the same issues with a dozen forks of the same program, some programs using Stockfish Networks some not. BUT, CCC does not claim to be this great bastion of competitive play, the highest quality tournament in the land. They are just a venue that plays games, so thats alright.

TCEC has been nothing more than a glorified basement tourney for years— not too different than anyone can run. The right way to do it , it to get the "authentic" authors together , decide what the rules and which engines are in . It would not hurt to have some "retired chess programers" in as well to provide some guidance. But the horse has probably already left the barn.

Geez...I certainly do not miss the constant clone accusations of the past from "authentic" "retired programmers".
No wonder major innovations come from other game programers: Go brought us MCTSs+NNs, even our brothers from shogi usurped us with NNUE when I thought that optimization is the forte of chess programmers. What we need is less of the cut-throat competition and clone accusations, and more of creating an atmosphere that encourages innovations.
No freaking ICGA like committee please.

[jdart]

I use my own labeled position sets. I have shared some of them. It is readily possible to generate a lot of positions for training. Generating labeled positions is expensive though. I have spent a few weeks of computer time with >100 cores to do that. That isn't possible for all programmers. So allowing re-use of labeled position sets seems to me to be a reasonable thing. Still, TCEC like any competition can make their own rules. I personally haven't had any problem with them. I think it is a little weird that they insisit on running on very big hardware when most engines don't have good scaling up to the number of cores they are using. 128 cores is not 2x better than 64, which is not 2x better than 32. But that's another issue.

--Jon

[Daniel Shawul]

I use my own labeled position sets. I have shared some of them. It is readily possible to generate a lot of positions for training. Generating labeled positions is expensive though. I have spent a few weeks of computer time with >100 cores to do that. That isn't possible for all programmers. So allowing re-use of labeled position sets seems to me to be a reasonable thing. Still, TCEC like any competition can make their own rules. I personally haven't had any problem with them. I think it is a little weird that they insisit on running on very big hardware when most engines don't have good scaling up to the number of cores they are using. 128 cores is not 2x better than 64, which is not 2x better than 32. But that's another issue.

--Jon

Jon, I hope you understand that I don't mean to single you out or anything. I actually happen to have your label.py in tune/ folder which I used in the past to lable epd files. Of course, It is ridiculous to say don't use labeled epds/pgns for training. If your engine is very weak, and you only use your evals to train it, it is going to take much more iterations of reinforcement learning, so it makes sense to learn from a stronger engine in one step.

[jdart]

Btw. also it is known for a long time (pre Alpha Zero) that it's possible to use learning to in effect reverse-engineer the evaluation of an engine, even a closed source one. It hasn't been a prominently featured issue before but it was known. Now that learning is pretty much a universal feature of strong engines it is more of an issue I guess. You think about it, though, don't human players learn from their stronger opponents, as well as now from the silicon monsters?

--Jon

[Dann Corbit]

Btw. also it is known for a long time (pre Alpha Zero) that it's possible to use learning to in effect reverse-engineer the evaluation of an engine, even a closed source one. It hasn't been a prominently featured issue before but it was known. Now that learning is pretty much a universal feature of strong engines it is more of an issue I guess. You think about it, though, don't human players learn from their stronger opponents, as well as now from the silicon monsters?

--Jon

when did we cross the invisible line from "learning from others is good" to "learning from others is bad"?

[Madeleine Birchfield]

Btw. also it is known for a long time (pre Alpha Zero) that it's possible to use learning to in effect reverse-engineer the evaluation of an engine, even a closed source one. It hasn't been a prominently featured issue before but it was known. Now that learning is pretty much a universal feature of strong engines it is more of an issue I guess. You think about it, though, don't human players learn from their stronger opponents, as well as now from the silicon monsters?

--Jon

TCEC's new rules regarding data usage in training is like banning Magnus Carlsen from the World Chess Championship in 2013 because he once studied a few games played by Vishy Anand.

[smatovic]

Hmm, how many Mega-Watt-hours went into the datasets of CCRL, chessdbcn, Lc0, etc.? How to catch up as independent developer without a CPU/GPU cluster? Fair competition vs. originality? Only the big players are able to pay the price for originality? Opening/sharing the datasets for free use and set up ways for own tuning, learning or implementation of NNs seems a way to go for me, or alike. There are plenty of varieties possible even with the same datasets or NNUE as architecture, imho.

--
Srdja

[AndrewGrant]

Hmm, how many Mega-Watt-hours went into the datasets of CCRL, chessdbcn, Lc0, etc.? How to catch up as independent developer without a CPU/GPU cluster? Fair competition vs. originality? Only the big players are able to pay the price for originality? Opening/sharing the datasets for free use and set up ways for own tuning, learning or implementation of NNs seems a way to go for me, or alike. There are plenty of varieties possible even with the same datasets or NNUE as architecture, imho.

Yes -- Exactly -- You've hit the nail on the head. I see two possible options, which are able to be consistent with their own rules.

[Option 1] Hold a Tournament where you force uniqueness and originality. Do this down to the bone. This means that engines cannot be using the Leela backends except for Leela. Engines cannot be using NNUE as of now except for Stockfish. Any Networks or even the main evaluation, need to be tuned on original data. This would make me happy, but its not going to happen. However, it is consistant. TCEC applies one set of rules for AlphaBeta engines, and another set for the GPU engines, due to their implicit $$$ bias.

[Option 2] Everything is up in the air. If you can provide an engine which has a large unique component, then it can compete. This means Allie can compete, as her search is different than Leela's, even if they used the same networks and back-ends. This means that all engines are free to use NNUE as a boost and share the same net. This means you can train on anything and everything. Its truly open, and allows unhinged innovation and building upon the work of others. Tournaments and Rating Lists would be responsible for figuring out what they believe is special enough to run.

You can even do both; have two tournaments. If I was running CCC or something, I would have the two events for each "season". Chesscom's Engine Tournament (OPEN), and Chesscom's Engine Tournament (CLOSED). This lets people dabble in using all of the latest and greatest, but also sets aside a special place for authors who are reinventing the wheel on their own, producing far more diverse chess entities. If style is a real concept, style exists in Option 1 but not Option 2.

What you cannot do, is what TCEC is doing. They are applying [Option 1] to the AlphaBeta engines in order to avoid letting multiple NNUEs enter causing Leela to fail to qualify for the SuperFinal (which she will, lol, get ready for S20), and then are applying [Option 2] to the GPU based Leela backends, because the want to increase the number of Leela-forks in order to maintain the viewer base that people like King Crusher have brought to the event.

Heres a question for people to ponder. Martin, the original founder of TCEC, was down in the dirt a computer chess fan and enthusiast. How come the people running the show now have to clue how engines work? Somehow TCEC got co-opted for profit, and stopped being about engine developers.

[smatovic]

...
[Option 2] Everything is up in the air. If you can provide an engine which has a large unique component, then it can compete.
...

I guess this is what TCEC is up to, afaik TCEC is per invitation only, so they have already their own bias of what is unique and what not, further they listen to the crowd (and maybe to one or two commercial units) to please the viewers...a fair contest between programmers vs. a spectacle for the crowd with a pinch of possible commercial interest, it is a mix of all...

--
Srdja

[Rebel]

Hmm, how many Mega-Watt-hours went into the datasets of CCRL, chessdbcn, Lc0, etc.? How to catch up as independent developer without a CPU/GPU cluster? Fair competition vs. originality? Only the big players are able to pay the price for originality? Opening/sharing the datasets for free use and set up ways for own tuning, learning or implementation of NNs seems a way to go for me, or alike. There are plenty of varieties possible even with the same datasets or NNUE as architecture, imho.

Yes -- Exactly -- You've hit the nail on the head. I see two possible options, which are able to be consistent with their own rules.

[Option 1] Hold a Tournament where you force uniqueness and originality. Do this down to the bone. This means that engines cannot be using the Leela backends except for Leela. Engines cannot be using NNUE as of now except for Stockfish. Any Networks or even the main evaluation, need to be tuned on original data. This would make me happy, but its not going to happen. However, it is consistant. TCEC applies one set of rules for AlphaBeta engines, and another set for the GPU engines, due to their implicit $$$ bias.

Stockfish has no exclusive rights on NNUE. They did not invent it, they took it. The NNUE concept is free for everybody.

[Option 2] Everything is up in the air. If you can provide an engine which has a large unique component, then it can compete. This means Allie can compete, as her search is different than Leela's, even if they used the same networks and back-ends. This means that all engines are free to use NNUE as a boost and share the same net. This means you can train on anything and everything. Its truly open, and allows unhinged innovation and building upon the work of others. Tournaments and Rating Lists would be responsible for figuring out what they believe is special enough to run.

Yep.

And that (the blue) will give trouble enough. This NNUE development will not go away, the cat is out of the bag, originality and fair competition is under pressure and that will be a hard nut to crack. One way (on the assumption if it's possible in theory) would be a tool that measures the similarity between nets, the question is more if that's a desirable initiative.

Other than that, be more creative than the rest, I am confident the current NNUE approach can be improved.

You can even do both; have two tournaments. If I was running CCC or something, I would have the two events for each "season". Chesscom's Engine Tournament (OPEN), and Chesscom's Engine Tournament (CLOSED). This lets people dabble in using all of the latest and greatest, but also sets aside a special place for authors who are reinventing the wheel on their own, producing far more diverse chess entities. If style is a real concept, style exists in Option 1 but not Option 2.

What you cannot do, is what TCEC is doing. They are applying [Option 1] to the AlphaBeta engines in order to avoid letting multiple NNUEs enter causing Leela to fail to qualify for the SuperFinal (which she will, lol, get ready for S20), and then are applying [Option 2] to the GPU based Leela backends, because the want to increase the number of Leela-forks in order to maintain the viewer base that people like King Crusher have brought to the event.

Heres a question for people to ponder. Martin, the original founder of TCEC, was down in the dirt a computer chess fan and enthusiast. How come the people running the show now have to clue how engines work? Somehow TCEC got co-opted for profit, and stopped being about engine developers.