Stockfish 14 release round the corner

[Modern Times]

I am not aware of any internal discussions in CCRL about Stockfish 14. It doesn't exist yet.

Except Stockfish 14 already exists:

https://stockfishchess.org/blog/2021/stockfish-14/

OK, then it exists as of today.

[Modern Times]

First, the collaboration with the Leela Chess Zero team - announced previously - has come to fruition. The LCZero team has provided a collection of billions of positions evaluated by Leela that we have combined with billions of positions evaluated by Stockfish to train the NNUE net that powers Stockfish 14. The fact that we could use and combine these datasets freely was essential for the progress made and demonstrates the power of open source and open data.

So this is a hybrid. It is trained on both Lc0 games as well as its own. That is very different from being trained purely on Lc0 games.

[Madeleine Birchfield]

First, the collaboration with the Leela Chess Zero team - announced previously - has come to fruition. The LCZero team has provided a collection of billions of positions evaluated by Leela that we have combined with billions of positions evaluated by Stockfish to train the NNUE net that powers Stockfish 14. The fact that we could use and combine these datasets freely was essential for the progress made and demonstrates the power of open source and open data.

So this is a hybrid. It is trained on both Lc0 games as well as its own. That is very different from being trained purely on Lc0 games.

In which case, there should be no problem, and CCRL should be able to test Stockfish 14.

[connor_mcmonigle]

By 2. I mean that you described an easy way to make every engine's net original, by your standards.

Also I have nothing against what you're claiming regarding to what works with NNUE training or doesn't. However I prefer following results rather than claims in my research, especially if it involves large resource requirements.

As for 1, a small experiment with a few 100m positions would be a good way to test.

This discussion has deviated a great deal from the original discussion, haha. Anyways, 100M positions is far from representative in my opinion. What works best with a 100M position dataset is significantly different from what works best with a 100G position dataset. I believe that with increasing scale, noisier data is actually desirable. The dynamics are very complicated and not well understood in any case

A fair point, but then we could never run any experiments.

For sure. I wasn't saying this shouldn't be tested. However, it will be impossible to match the scale of the Lc0 self play data already available (or very impractical anyways).

[Modern Times]

In which case, there should be no problem, and CCRL should be able to test Stockfish 14.

Yes that is my view.

[carldaman]

From my point of view, I don't see any reason why CCRL would not test Stockfish 14. In fact the CCRL chess960 list already includes a version that uses an NNUE based on Lc0 games. Individual testers will make their own choice with this one I think. It is a bit of a minefield, of course you then open the floodgates - other engines may use the Lc0 games, and then we have to defend why we would or would not test those. It is a lose-lose situation. You're damned if you do and damned if you don't.

I agree SF14 should be tested - pretty much goes without saying. One criterion for automatic inclusion should be that there be clear indication of the entity being the strongest on the given hardware, with no violation of any relevant license.

In short, if anyone can produce the strongest chess entity without violating anything, it should be allowed.

[Graham Banks]

Graham Banks has specifically said that CCRL will not be testing Stockfish 14 due to the use of Leela data:

Stockfish 14 may be released in couple of days. It is likely to have an 30-35 elo gain over SF 13.
SF14 with NNUE net trained from Lc0 training games. One gets to see best of both worlds!!

Won't that pose an issue for testing groups?
At present, we only test NNUE engines that have nets trained on their own games.

I didn't say that we wouldn't test it.
I was pointing out possible inconsistencies, but those would appear to be unfounded judging by the reactions.

By the way, I am only one CCRL tester. One person can't make decisions for the group.

[Graham Banks]

Graham is known for his hypocricy...

That's not very nice, Damir.

[Raphexon]

Graham Banks has specifically said that CCRL will not be testing Stockfish 14 due to the use of Leela data:

Stockfish 14 may be released in couple of days. It is likely to have an 30-35 elo gain over SF 13.
SF14 with NNUE net trained from Lc0 training games. One gets to see best of both worlds!!

Won't that pose an issue for testing groups?
At present, we only test NNUE engines that have nets trained on their own games.

I didn't say that we wouldn't test it.
I was pointing out possible inconsistencies, but those would appear to be unfounded judging by the reactions.

By the way, I am only one CCRL tester. One person can't make decisions for the group.

Do you agree that from here on out SF14 should listed as a Lc0 derivative?

[kranium]

"The fact that we could use and combine these datasets freely was essential for the progress made and demonstrates the power of open source and open data."

It's clear that both the LC0 team and the SF team are in complete agreement on this...
They believe that the data produced by engines, in whatever form should be open and free, and that this is key to progress.

We should not allow rating lists personal and tour directors to be pressured and/or bullied into decisions concerning rating list or tournament inclusion/exclusion.

Strict regulation and a policy of intense exclusion is the enemy of progress...