Is this SF NN almost like 20 MB book?

[Raphexon]

Just NN eval definitely isn't an opening book.


But what about policy head?
I definitely consider it bookish, Leela being very deterministic also supports it.

[dkappe]

Oh boy. Most nnue are trained at the beginning without regard to game outcome, often at depth 8. Most of the positions they see are maybe 18 to 19 ply into the game and later. They are in essence an approximation of an engine eval at d8.

You can look at my Toga and Night Nurse (based on Bad Gyal) nets and compare them to each other and the many stockfish nets, then explain to me how they are memorizing openings.

The Toga net in particular starts every game with a 19 ply random walk, at which point it revs up the engine. Every so often you’ll have enough noodling to get something like a KID, but that’s pure chance.

Why does every file large enough to contain an opening book have to a priori be an opening book?

[dkappe]

Just NN eval definitely isn't an opening book.


But what about policy head?
I definitely consider it bookish, Leela being very deterministic also supports it.

NNUE just outputs a single eval. No policy.

Also:
https://github.com/dkappe/leela-chess-w ... k-nonsense

[Raphexon]

Just NN eval definitely isn't an opening book.


But what about policy head?
I definitely consider it bookish, Leela being very deterministic also supports it.

NNUE just outputs a single eval. No policy.

Also:
https://github.com/dkappe/leela-chess-w ... k-nonsense

I wasn't talking NNUE when talking policyhead.

[Twipply]

https://github.com/dkappe/leela-chess-w ... k-nonsense

Is this meant to be a fair analysis of the idea presented?

"Leela is an opening book nonsense" The title you chose is far from impartial, which I can't say I like. It makes you sound like you've come into this already thinking you know what the answer is. If I think your analysis is bias or insulting, I'm likely not going to be swayed by it even if you happen to be right.

"I decided to test this opening book hypothesis with one of my favorite nets." What network is your favourite has absolutely no importance here. You should have done an analysis using as many networks as you could, and compared them to a variety of traditional engines, which you didn't do at all.

"I investigated a few opening lines" Looking at three openings is nowhere near comprehensive enough to draw a meaningful conclusion from.

"Good evidence that this net has memorized opening moves?" Not even remotely.

"Not so fast." You're putting words into the mouths of the readers, don't do that - especially not when you're wrong about what they're thinking.

"This is Little Ender" You managed to write a topic about Leela and then used an entirely different network in your attempt at an analysis.


You haven't shown that Leela doesn't contain an opening book, and you haven't shown that engines can't memorise openings.

[Milos]

...
Some tournaments disallow "books" but allow "neural networks", even if this distinction does not exist in reality, because you can train a neural network to remember openings.

But the most of chess engine can not "read" neural net, so they need common opening book.

So I encode the book for a classical engine in a proprietary format and call it a network. What's the difference?
Since A0 appeared I called it cheating that some engines can use a network of hundreds of MB in size while others can't use their books. It's rather silly.

[Milos]

Oh boy. Most nnue are trained at the beginning without regard to game outcome, often at depth 8. Most of the positions they see are maybe 18 to 19 ply into the game and later. They are in essence an approximation of an engine eval at d8.

And how is that different to Cerebellum book of Brainfish?
It's a "network" that gives you best move "trained" on SF engine eval of a very high depth. The only real difference to NNUE is that in Cerrebellum the search portion that SF performs is actually also encoded in the book.

[dkappe]

https://github.com/dkappe/leela-chess-w ... k-nonsense

Is this meant to be a fair analysis of the idea presented?

I’m sorry I hurt your feelings. (BTW, I found your engine to be an excellent sparring partner during the development of a0lite.)


It was entirely predictable that I would try my 18 piece or fewer approach with nnue’s.

Here is an nnue that has never seen a position with more than 18 pieces playing against itself, no book. I’m not sure it will be better at endgames, like ender. But it seems to know it’s way around the Spanish.

[pgn]

[Event "?"]
[Site "?"]
[Date "2020.08.04"]
[Round "1"]
[White "18pcB"]
[Black "18pcA"]
[Result "1-0"]
[ECO "C69"]
[GameDuration "00:04:49"]
[GameEndTime "2020-08-04T16:24:06.125 CDT"]
[GameStartTime "2020-08-04T16:19:16.591 CDT"]
[Opening "Ruy Lopez"]
[PlyCount "183"]
[Termination "adjudication"]
[TimeControl "60+1"]
[Variation "Exchange Variation , 5.O-O"]

1. e4 {+0.49/19 2.8s} e5 {-0.49/19 3.3s} 2. Nf3 {+0.55/17 0.89s}
Nc6 {-0.27/18 1.2s} 3. Bb5 {+0.34/19 2.0s} a6 {-0.32/22 3.0s}
4. Bxc6 {+0.26/21 2.4s} dxc6 {-0.30/18 0.68s} 5. O-O {+0.27/18 0.98s}
Qf6 {-0.26/17 1.1s} 6. d4 {+0.37/21 2.8s} exd4 {-0.33/18 1.2s}
7. Bg5 {+0.33/20 1.1s} Qd6 {-0.17/20 1.1s} 8. Nxd4 {+0.43/21 1.4s}
Be7 {-0.17/20 1.4s} 9. Be3 {+0.16/26 10s} Nf6 {-0.16/21 3.4s}
10. f3 {+0.43/17 1.1s} c5 {-0.08/18 0.87s} 11. Ne2 {+0.21/19 2.7s}
Be6 {+0.19/17 1.2s} 12. Nf4 {0.00/20 5.5s} Bc4 {+0.07/21 2.9s}
13. Qxd6 {0.00/18 0.80s} cxd6 {-0.10/21 2.0s} 14. Rf2 {+0.16/19 2.2s}
g5 {-0.39/20 3.7s} 15. Nd3 {+0.18/18 1.7s} Nd7 {-0.27/17 1.3s}
16. Nc3 {+0.26/17 1.1s} O-O {-0.29/20 7.6s} 17. Rd1 {+0.26/21 7.9s}
Be6 {-0.27/21 4.5s} 18. Rfd2 {+0.45/21 1.7s} b5 {-0.55/20 1.1s}
19. b3 {+0.33/19 2.2s} Rad8 {-0.20/17 1.2s} 20. a3 {+0.75/19 4.1s}
Kg7 {-0.34/21 5.8s} 21. Nf2 {+1.09/19 1.2s} Nb6 {-0.84/20 2.8s}
22. Nh1 {+0.90/19 1.6s} Rd7 {-0.45/18 1.1s} 23. g4 {+0.83/20 1.4s}
Rfd8 {-0.62/21 4.0s} 24. Ng3 {+1.06/18 1.1s} Kg6 {-0.69/18 0.80s}
25. h4 {+1.29/18 1.3s} h6 {-0.74/22 1.8s} 26. Nf5 {+1.04/22 3.0s}
Bf8 {-0.90/21 2.7s} 27. h5+ {+1.01/22 1.6s} Kh7 {-0.81/21 1.2s}
28. Bf2 {+1.12/20 1.1s} Bg7 {-0.87/22 2.5s} 29. Nxg7 {+0.87/22 2.9s}
Kxg7 {-0.66/20 1.9s} 30. Rd3 {+0.85/24 4.2s} f6 {-0.75/22 2.2s}
31. R3d2 {+0.78/20 1.1s} Kf7 {-0.77/23 5.9s} 32. Ne2 {+0.86/22 2.7s}
Nc8 {-0.78/25 5.0s} 33. Ng3 {+0.97/21 2.9s} Rc7 {-0.92/23 4.1s}
34. Nf1 {+1.22/20 1.9s} c4 {-0.83/23 1.6s} 35. b4 {+1.15/18 0.69s}
Rcd7 {-0.53/23 1.3s} 36. Ne3 {+1.00/22 1.00s} Ne7 {-1.07/24 1.9s}
37. Nf5 {+1.15/21 1.2s} Bxf5 {-0.66/20 0.60s} 38. gxf5 {+1.12/23 1.2s}
d5 {-0.94/23 2.8s} 39. Bc5 {+1.17/22 2.6s} c3 {-0.98/22 1.0s}
40. Rd3 {+1.79/22 0.64s} Nc6 {-0.89/23 1.0s} 41. Rxd5 {+1.79/22 0.89s}
Rxd5 {-1.76/21 1.3s} 42. exd5 {+1.71/23 1.3s} Ne5 {-1.45/21 0.61s}
43. Kf2 {+2.18/19 0.72s} Nc4 {-1.61/20 0.51s} 44. d6 {+2.59/20 0.94s}
Rd7 {-2.88/24 4.5s} 45. Rd3 {+2.79/20 0.81s} Rd8 {-2.91/20 0.50s}
46. Rxc3 {+2.85/21 0.86s} Ke8 {-2.95/22 0.78s} 47. Rd3 {+3.87/20 0.89s}
Kd7 {-3.20/20 0.80s} 48. Bd4 {+3.68/23 1.5s} Re8 {-3.06/19 0.24s}
49. Bxf6 {+3.65/23 0.78s} Rf8 {-3.46/22 0.78s} 50. Bg7 {+4.13/22 0.88s}
Rxf5 {-3.99/21 1.0s} 51. Kg3 {+4.63/23 1.2s} Rf7 {-3.97/21 0.72s}
52. Bxh6 {+4.64/21 1.3s} Rh7 {-3.57/18 0.49s} 53. Bxg5 {+4.98/21 0.82s}
Rxh5 {-3.82/20 1.0s} 54. Be7 {+4.99/21 1.4s} Rh1 {-4.69/21 3.8s}
55. f4 {+5.24/20 1.1s} Rf1 {-5.09/19 0.56s} 56. Kg4 {+5.25/19 0.88s}
Rf2 {-5.12/19 0.56s} 57. Rc3 {+5.65/21 1.3s} Rg2+ {-5.15/21 0.68s}
58. Kf5 {+6.05/23 2.2s} Rf2 {-4.60/19 0.38s} 59. Kg5 {+6.08/20 0.71s}
Ke6 {-5.80/18 0.90s} 60. Kg4 {+6.20/19 1.1s} Rg2+ {-5.57/20 2.6s}
61. Kf3 {+6.33/20 0.67s} Rg1 {-4.39/19 0.54s} 62. f5+ {+6.90/18 0.70s}
Kd7 {-4.46/17 0.38s} 63. f6 {+6.80/19 1.1s} Ke6 {-6.81/20 2.7s}
64. Kf2 {+7.23/21 0.87s} Rg6 {-5.35/15 0.12s} 65. d7 {+7.44/19 0.71s}
Kxd7 {-7.23/18 0.72s} 66. Rf3 {+7.89/21 1.4s} Ne5 {-6.91/17 0.26s}
67. f7 {+8.03/19 0.68s} Nxf7 {-7.66/19 0.99s} 68. Rxf7 {+8.19/20 0.97s}
Ke6 {-7.52/18 1.1s} 69. Rh7 {+8.35/20 1.1s} Kd5 {-8.06/17 0.60s}
70. Rh4 {+8.36/19 0.99s} Ke6 {-8.74/19 3.1s} 71. Bc5 {+9.04/18 0.81s}
Kf7 {-9.32/19 1.4s} 72. Ke2 {+9.55/18 0.77s} Kg8 {-9.97/16 0.93s}
73. Rh5 {+10.44/22 1.8s} Kf7 {-10.07/19 0.60s} 74. Kd3 {+10.74/19 0.72s}
Rf6 {-10.08/20 0.94s} 75. Ke4 {+11.22/19 0.68s} Rg6 {-10.09/18 0.63s}
76. Rh7+ {+11.69/19 0.82s} Kf6 {-11.78/19 1.8s} 77. Be3 {+12.20/18 0.73s}
Rg8 {-11.78/18 0.27s} 78. Rh6+ {+12.81/17 0.81s} Ke7 {-12.94/18 1.7s}
79. Rxa6 {+14.62/20 1.3s} Kd7 {-12.49/17 0.51s} 80. Kd5 {+14.59/19 0.72s}
Kc7 {-15.12/20 1.5s} 81. Kc5 {+16.64/19 0.76s} Re8 {-10.56/17 0.38s}
82. Bd4 {+18.86/19 1.2s} Re2 {-20.31/18 1.6s} 83. c3 {+21.45/18 0.77s}
Re4 {-22.45/18 1.1s} 84. Kxb5 {+22.10/17 0.80s} Rh4 {-15.62/15 0.28s}
85. a4 {+29.19/19 0.94s} Kd7 {-29.87/17 1.2s} 86. a5 {+32.67/18 0.81s}
Rh1 {-38.97/17 1.4s} 87. Rb6 {+154.08/25 2.5s} Rh5+ {-154.00/16 0.70s}
88. Bc5 {+154.09/30 0.60s} Ke8 {-154.10/20 0.34s} 89. a6 {+M23/28 0.78s}
Rh1 {-M22/26 0.43s} 90. a7 {+M21/32 0.81s} Ra1 {-M18/25 0.37s}
91. Ra6 {+M19/33 0.81s} Rxa6 {-M16/31 0.39s}
92. Kxa6 {+M13/37 0.79s, White wins by adjudication: SyzygyTB} 1-0
[/pgn]

[Milos]

https://github.com/dkappe/leela-chess-w ... k-nonsense

Is this meant to be a fair analysis of the idea presented?

I’m sorry I hurt your feelings. (BTW, I found your engine to be an excellent sparring partner during the development of a0lite.)


It was entirely predictable that I would try my 18 piece or fewer approach with nnue’s.

Here is an nnue that has never seen a position with more than 18 pieces playing against itself, no book. I’m not sure it will be better at endgames, like ender. But it seems to know it’s way around the Spanish.

[pgn]

[Event "?"]
[Site "?"]
[Date "2020.08.04"]
[Round "1"]
[White "18pcB"]
[Black "18pcA"]
[Result "1-0"]
[ECO "C69"]
[GameDuration "00:04:49"]
[GameEndTime "2020-08-04T16:24:06.125 CDT"]
[GameStartTime "2020-08-04T16:19:16.591 CDT"]
[Opening "Ruy Lopez"]
[PlyCount "183"]
[Termination "adjudication"]
[TimeControl "60+1"]
[Variation "Exchange Variation , 5.O-O"]

1. e4 {+0.49/19 2.8s} e5 {-0.49/19 3.3s} 2. Nf3 {+0.55/17 0.89s}
Nc6 {-0.27/18 1.2s} 3. Bb5 {+0.34/19 2.0s} a6 {-0.32/22 3.0s}
4. Bxc6 {+0.26/21 2.4s} dxc6 {-0.30/18 0.68s} 5. O-O {+0.27/18 0.98s}
Qf6 {-0.26/17 1.1s} 6. d4 {+0.37/21 2.8s} exd4 {-0.33/18 1.2s}
7. Bg5 {+0.33/20 1.1s} Qd6 {-0.17/20 1.1s} 8. Nxd4 {+0.43/21 1.4s}
Be7 {-0.17/20 1.4s} 9. Be3 {+0.16/26 10s} Nf6 {-0.16/21 3.4s}
10. f3 {+0.43/17 1.1s} c5 {-0.08/18 0.87s} 11. Ne2 {+0.21/19 2.7s}
Be6 {+0.19/17 1.2s} 12. Nf4 {0.00/20 5.5s} Bc4 {+0.07/21 2.9s}
13. Qxd6 {0.00/18 0.80s} cxd6 {-0.10/21 2.0s} 14. Rf2 {+0.16/19 2.2s}
g5 {-0.39/20 3.7s} 15. Nd3 {+0.18/18 1.7s} Nd7 {-0.27/17 1.3s}
16. Nc3 {+0.26/17 1.1s} O-O {-0.29/20 7.6s} 17. Rd1 {+0.26/21 7.9s}
Be6 {-0.27/21 4.5s} 18. Rfd2 {+0.45/21 1.7s} b5 {-0.55/20 1.1s}
19. b3 {+0.33/19 2.2s} Rad8 {-0.20/17 1.2s} 20. a3 {+0.75/19 4.1s}
Kg7 {-0.34/21 5.8s} 21. Nf2 {+1.09/19 1.2s} Nb6 {-0.84/20 2.8s}
22. Nh1 {+0.90/19 1.6s} Rd7 {-0.45/18 1.1s} 23. g4 {+0.83/20 1.4s}
Rfd8 {-0.62/21 4.0s} 24. Ng3 {+1.06/18 1.1s} Kg6 {-0.69/18 0.80s}
25. h4 {+1.29/18 1.3s} h6 {-0.74/22 1.8s} 26. Nf5 {+1.04/22 3.0s}
Bf8 {-0.90/21 2.7s} 27. h5+ {+1.01/22 1.6s} Kh7 {-0.81/21 1.2s}
28. Bf2 {+1.12/20 1.1s} Bg7 {-0.87/22 2.5s} 29. Nxg7 {+0.87/22 2.9s}
Kxg7 {-0.66/20 1.9s} 30. Rd3 {+0.85/24 4.2s} f6 {-0.75/22 2.2s}
31. R3d2 {+0.78/20 1.1s} Kf7 {-0.77/23 5.9s} 32. Ne2 {+0.86/22 2.7s}
Nc8 {-0.78/25 5.0s} 33. Ng3 {+0.97/21 2.9s} Rc7 {-0.92/23 4.1s}
34. Nf1 {+1.22/20 1.9s} c4 {-0.83/23 1.6s} 35. b4 {+1.15/18 0.69s}
Rcd7 {-0.53/23 1.3s} 36. Ne3 {+1.00/22 1.00s} Ne7 {-1.07/24 1.9s}
37. Nf5 {+1.15/21 1.2s} Bxf5 {-0.66/20 0.60s} 38. gxf5 {+1.12/23 1.2s}
d5 {-0.94/23 2.8s} 39. Bc5 {+1.17/22 2.6s} c3 {-0.98/22 1.0s}
40. Rd3 {+1.79/22 0.64s} Nc6 {-0.89/23 1.0s} 41. Rxd5 {+1.79/22 0.89s}
Rxd5 {-1.76/21 1.3s} 42. exd5 {+1.71/23 1.3s} Ne5 {-1.45/21 0.61s}
43. Kf2 {+2.18/19 0.72s} Nc4 {-1.61/20 0.51s} 44. d6 {+2.59/20 0.94s}
Rd7 {-2.88/24 4.5s} 45. Rd3 {+2.79/20 0.81s} Rd8 {-2.91/20 0.50s}
46. Rxc3 {+2.85/21 0.86s} Ke8 {-2.95/22 0.78s} 47. Rd3 {+3.87/20 0.89s}
Kd7 {-3.20/20 0.80s} 48. Bd4 {+3.68/23 1.5s} Re8 {-3.06/19 0.24s}
49. Bxf6 {+3.65/23 0.78s} Rf8 {-3.46/22 0.78s} 50. Bg7 {+4.13/22 0.88s}
Rxf5 {-3.99/21 1.0s} 51. Kg3 {+4.63/23 1.2s} Rf7 {-3.97/21 0.72s}
52. Bxh6 {+4.64/21 1.3s} Rh7 {-3.57/18 0.49s} 53. Bxg5 {+4.98/21 0.82s}
Rxh5 {-3.82/20 1.0s} 54. Be7 {+4.99/21 1.4s} Rh1 {-4.69/21 3.8s}
55. f4 {+5.24/20 1.1s} Rf1 {-5.09/19 0.56s} 56. Kg4 {+5.25/19 0.88s}
Rf2 {-5.12/19 0.56s} 57. Rc3 {+5.65/21 1.3s} Rg2+ {-5.15/21 0.68s}
58. Kf5 {+6.05/23 2.2s} Rf2 {-4.60/19 0.38s} 59. Kg5 {+6.08/20 0.71s}
Ke6 {-5.80/18 0.90s} 60. Kg4 {+6.20/19 1.1s} Rg2+ {-5.57/20 2.6s}
61. Kf3 {+6.33/20 0.67s} Rg1 {-4.39/19 0.54s} 62. f5+ {+6.90/18 0.70s}
Kd7 {-4.46/17 0.38s} 63. f6 {+6.80/19 1.1s} Ke6 {-6.81/20 2.7s}
64. Kf2 {+7.23/21 0.87s} Rg6 {-5.35/15 0.12s} 65. d7 {+7.44/19 0.71s}
Kxd7 {-7.23/18 0.72s} 66. Rf3 {+7.89/21 1.4s} Ne5 {-6.91/17 0.26s}
67. f7 {+8.03/19 0.68s} Nxf7 {-7.66/19 0.99s} 68. Rxf7 {+8.19/20 0.97s}
Ke6 {-7.52/18 1.1s} 69. Rh7 {+8.35/20 1.1s} Kd5 {-8.06/17 0.60s}
70. Rh4 {+8.36/19 0.99s} Ke6 {-8.74/19 3.1s} 71. Bc5 {+9.04/18 0.81s}
Kf7 {-9.32/19 1.4s} 72. Ke2 {+9.55/18 0.77s} Kg8 {-9.97/16 0.93s}
73. Rh5 {+10.44/22 1.8s} Kf7 {-10.07/19 0.60s} 74. Kd3 {+10.74/19 0.72s}
Rf6 {-10.08/20 0.94s} 75. Ke4 {+11.22/19 0.68s} Rg6 {-10.09/18 0.63s}
76. Rh7+ {+11.69/19 0.82s} Kf6 {-11.78/19 1.8s} 77. Be3 {+12.20/18 0.73s}
Rg8 {-11.78/18 0.27s} 78. Rh6+ {+12.81/17 0.81s} Ke7 {-12.94/18 1.7s}
79. Rxa6 {+14.62/20 1.3s} Kd7 {-12.49/17 0.51s} 80. Kd5 {+14.59/19 0.72s}
Kc7 {-15.12/20 1.5s} 81. Kc5 {+16.64/19 0.76s} Re8 {-10.56/17 0.38s}
82. Bd4 {+18.86/19 1.2s} Re2 {-20.31/18 1.6s} 83. c3 {+21.45/18 0.77s}
Re4 {-22.45/18 1.1s} 84. Kxb5 {+22.10/17 0.80s} Rh4 {-15.62/15 0.28s}
85. a4 {+29.19/19 0.94s} Kd7 {-29.87/17 1.2s} 86. a5 {+32.67/18 0.81s}
Rh1 {-38.97/17 1.4s} 87. Rb6 {+154.08/25 2.5s} Rh5+ {-154.00/16 0.70s}
88. Bc5 {+154.09/30 0.60s} Ke8 {-154.10/20 0.34s} 89. a6 {+M23/28 0.78s}
Rh1 {-M22/26 0.43s} 90. a7 {+M21/32 0.81s} Ra1 {-M18/25 0.37s}
91. Ra6 {+M19/33 0.81s} Rxa6 {-M16/31 0.39s}
92. Kxa6 {+M13/37 0.79s, White wins by adjudication: SyzygyTB} 1-0
[/pgn]

That proves absolutely nothing. SF with eval consisting only of material+mobility+PST also knows its way around spanish and hasn't seen any chess positon ever. That's thanks to search not thanks to eval.
How bad is NNUE eval really one sees when testing SF depth 5 with minimum eval (material+mobility+PST) vs SF-NNUE depth 1. It's a slaughter house.

[Twipply]

I’m sorry I hurt your feelings.

I reacted strongly not because of feelings, but because I think this topic has basically invalidated some of the more recent TCEC Superfinal results, and the admins there should stop ignoring it. However, even if my feelings were hurt, that would not invalidate what I've said nor would it validate your post.

(BTW, I found your engine to be an excellent sparring partner during the development of a0lite.)

Thanks. I'm glad it worked well for you.