Dispelling the Myth of NNUE with LazySMP: An Analysis

[D Sceviour]

OK! Got it. I will look at that maybe in the new year. My project for today is creating a program to disassemble polyglot books. Somebody said that program does not exist. If not, it will exist shortly.

Look at the polyglot option dump-book. This may be what you want. Below some sample output.

Code: Select all

Dump of "performance.bin" for white.
1: 1. e4{33%} a6 2. d4{100%} b5 3. Nf3{64%} e6 4. Bd3{100%} Bb7 5. O-O{75%} c5 6. c3{100%} Nf6 7. Re1{100%} 
2: 1. e4{33%} a6 2. d4{100%} b5 3. Nf3{64%} e6 4. Bd3{100%} Bb7 5. Qe2{25%} 
3: 1. e4{33%} a6 2. d4{100%} b5 3. Nf3{64%} Bb7 4. Bd3{100%} e6 {trans: line=1, ply=8}
4: 1. e4{33%} a6 2. d4{100%} b5 3. Bd3{36%} Bb7 4. Nf3{100%} {trans: line=3, ply=7}
5: 1. e4{33%} a6 2. d4{100%} e6 3. Nf3{56%} b5 {trans: line=1, ply=6}
6: 1. e4{33%} a6 2. d4{100%} e6 3. Nf3{56%} c5 4. c3{100%} d5 5. e5{100%} Bd7 6. Bd3{100%} cxd4 7. Nxd4{100%} Nc6 8. Nxc6{100%} 
7: 1. e4{33%} a6 2. d4{100%} e6 3. Bd3{44%} 
8: 1. e4{33%} b6 2. d4{100%} e6 3. Nf3{57%} Bb7 4. Bd3{100%} c5 5. c3{87%} Nf6 6. Qe2{61%} cxd4 7. cxd4{100%} 
9: 1. e4{33%} b6 2. d4{100%} e6 3. Nf3{57%} Bb7 4. Bd3{100%} c5 5. c3{87%} Nf6 6. Qe2{61%} d5 7. e5{100%} 
10: 1. e4{33%} b6 2. d4{100%} e6 3. Nf3{57%} Bb7 4. Bd3{100%} c5 5. c3{87%} Nf6 6. Qe2{61%} Be7 7. O-O{100%} d5 8. e5{100%} Nfd7 9. a3{100%} 
11: 1. e4{33%} b6 2. d4{100%} e6 3. Nf3{57%} Bb7 4. Bd3{100%} c5 5. c3{87%} Nf6 6. Qe2{61%} Be7 7. O-O{100%} Nc6 8. a3{100%} 
12: 1. e4{33%} b6 2. d4{100%} e6 3. Nf3{57%} Bb7 4. Bd3{100%} c5 5. c3{87%} Nf6 6. Nbd2{22%} cxd4 7. cxd4{100%} 
13: 1. e4{33%} b6 2. d4{100%} e6 3. Nf3{57%} Bb7 4. Bd3{100%} c5 5. c3{87%} Nf6 6. Nbd2{22%} d5 7. e5{100%} 
14: 1. e4{33%} b6 2. d4{100%} e6 3. Nf3{57%} Bb7 4. Bd3{100%} c5 5. c3{87%} Nf6 6. Nbd2{22%} Nc6 7. a3{100%} d5 8. e5{100%} Nd7 9. O-O{100%} 
15: 1. e4{33%} b6 2. d4{100%} e6 3. Nf3{57%} Bb7 4. Bd3{100%} c5 5. c3{87%} Nf6 6. Nbd2{22%} Be7 7. O-O{100%} cxd4 8. cxd4{100%} Ba6 9. Bxa6{100%} Nxa6 10. Qe2{100%} Nc7 11. d5{100%} d6 12. dxe6{100%} 
16: 1. e4{33%} b6 2. d4{100%} e6 3. Nf3{57%} Bb7 4. Bd3{100%} c5 5. c3{87%} Nf6 6. e5{18%} 
17: 1. e4{33%} b6 2. d4{100%} e6 3. Nf3{57%} Bb7 4. Bd3{100%} c5 5. O-O{13%} cxd4 6. Nxd4{100%} a6 7. c4{100%} 
18: 1. e4{33%} b6 2. d4{100%} e6 3. Nf3{57%} Bb7 4. Bd3{100%} d6 5. O-O{68%} Nd7 6. c4{100%} 
19: 1. e4{33%} b6 2. d4{100%} e6 3. Nf3{57%} Bb7 4. Bd3{100%} d6 5. c4{32%} Nd7 6. Nc3{100%} 
20: 1. e4{33%} b6 2. d4{100%} e6 3. Nf3{57%} Bb7 4. Bd3{100%} g6 5. Bg5{100%} 
...
...

Yes, thank you. This is something like what I expected. dump-book appears in later versions of polyglot, I really have to keep myself updated on this new polyglot stuff. Schooner has growing internal polyglot utilities, so I might add my own version of this. It does not take a lot of code.

[D Sceviour]

@Andrew sorry for this offspring about polyglot book dump (started by Denis), may be the thread could be split up?

Andrews post should have been put in the programming section.

@Michel
No one ever answered my post about book dump not working correctly in most cases on Windows (at least).
Note that I am mostly interested in the first part not working (cut off lines from the dump file)

http://talkchess.com/forum3/viewtopic.p ... mp#p718824

I tried to compile Polyglot on my machine (latest snapshot from HGMs repo) today, but it gave me errors due to multiple definitions
of pg_header and pg_header magic IIRC.

Here is a very simple example for what is going on here with the dump file not written correctly:
It looks like Polyglot always already exits before the dump is fully written?

Any idea for a fix?

I can see two problems with a polyglot dump not completing. The first is that the book is composed of positional FEN fragments. There is no way to complete a line that starts in the middle of nowhere. I think this is the problem people are facing. How to disassemble a polyglot key and translate it into a position?

The second problem is file size. Try to compile for 64 bit otherwise the file sizes are limited by 32-bit constraints.

[Guenther]

@Andrew sorry for this offspring about polyglot book dump (started by Denis), may be the thread could be split up?

Andrews post should have been put in the programming section.

@Michel
No one ever answered my post about book dump not working correctly in most cases on Windows (at least).
Note that I am mostly interested in the first part not working (cut off lines from the dump file)

http://talkchess.com/forum3/viewtopic.p ... mp#p718824

I tried to compile Polyglot on my machine (latest snapshot from HGMs repo) today, but it gave me errors due to multiple definitions
of pg_header and pg_header magic IIRC.

Here is a very simple example for what is going on here with the dump file not written correctly:
It looks like Polyglot always already exits before the dump is fully written?

Any idea for a fix?

I can see two problems with a polyglot dump not completing. The first is that the book is composed of positional FEN fragments. There is no way to complete a line that starts in the middle of nowhere. I think this is the problem people are facing. How to disassemble a polyglot key and translate it into a position?

The second problem is file size. Try to compile for 64 bit otherwise the file sizes are limited by 32-bit constraints.

My problem has nothing to do with this. You can see it from the exact procedure posted.
(and the feature is already quite old too BTW)

[Gerd Isenberg]

While doing a lot of branchless avx2 computation versus all the conditional GP stuff of a classic eval - I would expect a decrease from HT improvement. Afaik HT profits most if threads have stalls due to branch misspredictions, dependency or cache misses, since the other hyper thread can keep the pipeline from sitting idle. So why does Ethereal have better HT based scaling compared to Stockfish and KomodoDragon? Maybe single Ethereal thread stalls more often? Layout of the weights, L1/L2 sharing issues?

[AndrewGrant]

That is simple to answer. No one was talking about your engine.
The issue was with Stockfish NNUE. The claims about faulty testing in CCRL. Was about Stickfish NNUE.

Okay good, we have officially moved the goal posts. We no longer believe that NNUE is the cause for engines to scale worse, but that it is the cause for Stockfish to scale worse. More tests coming.

[mwyoung]

That is simple to answer. No one was talking about your engine.
The issue was with Stockfish NNUE. The claims about faulty testing in CCRL. Was about Stickfish NNUE.

Okay good, we have officially moved the goal posts. We no longer believe that NNUE is the cause for engines to scale worse, but that it is the cause for Stockfish to scale worse. More tests coming.

You misunderstand. NNUE was a way of saying, It was not Classical Stockfish. Here is how Classical Stockfish behaves. Here is how NNUE behaves.

You putting NNUE into your engine does not make you Stockfish. This was a NNUE issue with Stockfish.

[AndrewGrant]

That is simple to answer. No one was talking about your engine.
The issue was with Stockfish NNUE. The claims about faulty testing in CCRL. Was about Stickfish NNUE.

Okay good, we have officially moved the goal posts. We no longer believe that NNUE is the cause for engines to scale worse, but that it is the cause for Stockfish to scale worse. More tests coming.

You misunderstand. NNUE was a way of saying, It was not Classical Stockfish. Here is how Classical Stockfish behaves. Here is how NNUE behaves.
You putting NNUE into your engine does not make you Stockfish. This was a NNUE issue with Stockfish.

Goal posts moved; Got it. Starting up Stockfish tests now. Komodo to follow.

[mwyoung]

Code: Select all

That is simple to answer. No one was talking about your engine.
The issue was with Stockfish NNUE. The claims about faulty testing in CCRL. Was about Stickfish NNUE.

Okay good, we have officially moved the goal posts. We no longer believe that NNUE is the cause for engines to scale worse, but that it is the cause for Stockfish to scale worse. More tests coming.

You misunderstand. NNUE was a way of saying, It was not Classical Stockfish. Here is how Classical Stockfish behaves. Here is how NNUE behaves.
You putting NNUE into your engine does not make you Stockfish. This was a NNUE issue with Stockfish.

Goal posts moved; Got it. Starting up Stockfish tests now. Komodo to follow.

Yep it moved. You think you are Stockfish NNUE. After trashing NNUE in this forum. You joined the club, and think you are now the best. Because you added NNUE. Like Stockfish.

Current TCEC.

Code: Select all

1	
Stockfish 20201225
15	10.5	70%	8 [6/2]	2 [0/2]	5 [1/4]	68	3592	4 [3596]
2	
LCZero v0.26.3_T60.66740
15	9.5	63.33%	7 [7/0]	3 [0/3]	5 [0/5]	66.75	3575	3 [3578]
3	
KomodoDragon 2647.00
15	9	60%	7 [7/0]	4 [0/4]	4 [1/3]	58.25	3522	37 [3559]
4	
Stoofvlees II a16
15	8.5	56.67%	5 [5/0]	3 [0/3]	7 [3/4]	58	3513	6 [3519]
5	
AllieStein v0.8-120f959_net-15.0
15	8	53.33%	4 [4/0]	3 [0/3]	8 [3/5]	55.25	3509	4 [3513]
6	
ScorpioNN 3.0.11
15	5.5	36.67%	1 [1/0]	5 [1/4]	9 [6/3]	37.75	3496	-7 [3489]
7	
Ethereal TCEC S20 DivP
15	5	33.33%	1 [1/0]	6 [0/6]	8 [7/1]	36.75	3482	-6 [3476]
8	
rofChade 2.309
15	4	26.67%	1 [1/0]	8 [1/7]

[AndrewGrant]

Yep it moved. You think you are Stockfish NNUE. After trashing NNUE in this forum. You joined the club, and think you are now the best. Because you added NNUE. Like Stockfish.

Absolutely and unequvically no. Ethereal NNUE is a piece of shit derivative program, and has no value in it. The same can be said about all NNUE derivatives of Stockfish. Especially the ones actually using Stockfishs code. Even though I wrote both the trainer and the implementation without SF as a reference, its still unoriginal crap that no chess engine enthusiast should use. I actively encourage, both here and in TCEC / CCC chat, anyone who cares about Ethereal use a non-NNUE version.

[AndrewGrant]

New data. I had took the current version of Stockfish commited to the repo, and compiled it using arch=x86-64-modern, which includes AVX and AVX2 optimizations. In order to keep the elo difference between the two versions (NNUE and Classical), I had to create time odds in favor of Classical. In these tests, the NNUE version recieved half of the time of the Classical version. This is to reduce the elo difference in order to get better estimations. The actual difference does not matter. We care only for the difference between them as a function of thread counts.

Code: Select all

Stockfish-NNUE-1C vs Stockfish-1C: 2340 -  909 - 2401 [0.627] 5650 + 90
Stockfish-NNUE-2C vs Stockfish-2C:  873 -  242 -  885 [0.658] 2000 +114
Stockfish-NNUE-4C vs Stockfish-4C:  516 -  141 -  593 [0.650] 1250 +108

Now both Ethereal and Stockfish have evidence suggesting that NNUE does not impact scaling in any meaningful way. In fact, due to the time control of my tests, I have seen no such elo loss due to increased threads, thus adding even more evidence to my original theory that the false claims about NNUE and LazySMP are about the unfortunate reality of the diminishing returns as chess nears draw death.

After running these sets, for both engines, I will claim now that NNUE does not exhibit meaningfully difference scaling. Which is sound, since no one in this thread has actually proposed a reason as to why the scaling was worse, they have only made what they believe to be an observation of worse scaling, and then attributed it to NNUE.

This closes the case here. Thanks for reading.