NNUE variation

[Rebel]

Added Orion_NNUE, Nemorino_NNUE and Rubichess_NNUE. Rubichess with one of the latests Sergio net.

http://rebel13.nl/dump/nnue-100ms.html
http://rebel13.nl/dump/nnue-250ms.html
http://rebel13.nl/dump/nnue-500ms.html
http://rebel13.nl/dump/nnue-1000ms.html

[Tony P.]

^ At 1000 ms, Minic's correlations are <=48.28%, whereas between any pair of other engines, they're >=54.77%. If I didn't know which engine had a net trained on a different dataset, I'd easily guess it from the table. (Sorry for the edit, I was looking at the 500 ms table initially; there, Minic <=45.84%, the rest >=52.08%. So 50% is a candidate for a cutoff for NNUE.)

[Albert Silver]

Tested several NNUE nets with SF12 and Minic (depth=1) and it seems to me there is a lot a variation between nets, even between the last 7 Sergio nets.

http://rebel13.nl/dump/sf12.html
http://rebel13.nl/dump/minic.html

Maybe this softens Andrew's pain a bit.

What worries me about neural nets (also Lc0) is that it changes your engine playing style while you are not aware of it. Oh wait, that already happens when you only look at the cute-chess results without ever replaying a game

With all due respect, the vast majority of players and engine developers are completely blind to questions of style in chess. And this is normal mind you. I have been more exposed to this topic than most, and discussed it with top players, and know that even my sensitivity to style is nothing compared to the best players who can pick up on even minute details in choice and move order. The reason is that the top players do more than just play the best moves they can. They play a metagame in which they try to enter the mind of their rival to anticipate not just the best move they perceive but also the reply their opponent is most likely to choose.

[xr_a_y]

Added Orion_NNUE, Nemorino_NNUE and Rubichess_NNUE. Rubichess with one of the latests Sergio net.

http://rebel13.nl/dump/nnue-100ms.html
http://rebel13.nl/dump/nnue-250ms.html
http://rebel13.nl/dump/nnue-500ms.html
http://rebel13.nl/dump/nnue-1000ms.html

Thanks for those tests with SV nets.
Can you please compare that to using a pure Minic net please ? (https://github.com/tryingsomestuff/NNUE ... trient.bin)

[Rebel]

^ At 1000 ms, Minic's correlations are <=48.28%, whereas between any pair of other engines, they're >=54.77%. If I didn't know which engine had a net trained on a different dataset, I'd easily guess it from the table. (Sorry for the edit, I was looking at the 500 ms table initially; there, Minic <=45.84%, the rest >=52.08%. So 50% is a candidate for a cutoff for NNUE.)

That's an important observation

Minic uses the nn-97f742aaefcd net from the SF NNUE page dated August 3 long before the SF12 release thus most likely a Sergio net which they renamed from bin to nnue. Which would indicate the Minic results are just an accident.

Accident or not, I will do 2 extra tests, 1) run Minic with another net, 2) run Rubichess with the Minic net and see if the pattern remains.

[Rebel]

Added Orion_NNUE, Nemorino_NNUE and Rubichess_NNUE. Rubichess with one of the latests Sergio net.

http://rebel13.nl/dump/nnue-100ms.html
http://rebel13.nl/dump/nnue-250ms.html
http://rebel13.nl/dump/nnue-500ms.html
http://rebel13.nl/dump/nnue-1000ms.html

Thanks for those tests with SV nets.
Can you please compare that to using a pure Minic net please ? (https://github.com/tryingsomestuff/NNUE ... trient.bin)

Will do that too.

[Tony P.]

Minic uses the nn-97f742aaefcd net from the SF NNUE page dated August 3 long before the SF12 release thus most likely a Sergio net which they renamed from bin to nnue. Which would indicate the Minic results are just an accident.

Accident or not, I will do 2 extra tests, 1) run Minic with another net, 2) run Rubichess with the Minic net and see if the pattern remains.

Thanks! Indeed,

nn-97f742aaefcd.nnue is 20200801-1515.bin

Btw, I suggest to add the release dates (at least MMDD) to the numeration of SV nets in SimEx tables, in order to sort them chronologically, otherwise it's harder to analyze temporal patterns.

What surprises me the most is the eval similarity between SF12 and Lc0, which suggests that Sergio's method of NNUE training ostensibly distills knowledge so well that SF NNUE is a marriage of a fast AB searcher with most of the knowledge captured by Leela T60 nets.

[xr_a_y]

As name of the nets used are sometimes not present in the tables, things can rapidly get hard to understand here.
I think, all the previous results about "Minic NNUE" where using a very good SV net.

Next will probably come a test with Minic using a net built using Minic data and learner (this net is more than 100Elo weaker than SV nets).

If with this "pure Minic net", similarity with SF12 is not well under the one when Minic is using SV net then it can mean that NNUE nets are converging in the same direction. This won't be crazy as most engine search algorithms are using the same tricks (more or less well tuned) and main eval patterns are also more or less the same.

[xr_a_y]

Maybe I missed something. What is "SF12 Minic_NNUE" ? is it SF12 using "Nascent Nutrient" ?

Please please clarify the naming convention in those results.

Those tests are great and deserved to be well readable.

[Rebel]

Okay, new results.

http://rebel13.nl/dump/nnue-100ms.html
http://rebel13.nl/dump/nnue-250ms.html
http://rebel13.nl/dump/nnue-500ms.html
http://rebel13.nl/dump/nnue-1000ms.html

1. The Minic 2.48_nascent_nutrient net hardly made a difference, same constant similarity.

2. Giving Minic 2.48 the official SF12 net the normal increasing similarity is in sync with other engines.

3. Giving Rubichess the Minic 2.48 net also follows the same pattern as the other engines.

Regarding net-name-calling, the Sergio net numbering was fine, now the gibberish SF numbering probably is going to be the new standard, I suggest the Igel way, for Minic -> Minic-248.nnue