LCzero network

[menniepals]

Which of the networks is the best?

[mehmet karaman]

For now 31970 is the best

31970- 11248: +9 -5 =186
Hardware used: 2080Ti and 6men TBs
Time control: 2min + 1sec
Book: Private (about 30 = positions)

31970- 11248: +39 - 27 =139
GPU 2080Ti and 6men TBs
5000 nodes/move

https://groups.google.com/forum/#!forum/lczero

[Guenther]

For now 31970 is the best

31970- 11248: +9 -5 =186
Hardware used: 2080Ti and 6men TBs
Time control: 2min + 1sec
Book: Private (about 30 = positions)

31970- 11248: +39 - 27 =139
GPU 2080Ti and 6men TBs
5000 nodes/move

https://groups.google.com/forum/#!forum/lczero

Are the games available? I have some doubts, because of the low number of start positions (what length btw and which selection criteria?).
If the games are not independent enough, the error bars become too high.

BTW you also did not mention that both test results are from different testers... for quoting a result it would be preferred,
having that information available too

[mehmet karaman]

31970 vs 11248: 39 - 27 - 139

https://groups.google.com/forum/#!topic ... MfLfOQmgn0

[Guenther]

There is also another effect, which makes the test quite unreliable, due to playing an instance of itself,
instead versus a variety of other programs, as done in established rating lists.

The two respectively different NNs play nearly identical moves, despite the very fast tc.
In the end the result just becomes a lottery. I just looked at the first two games and they show
up already as dupes for 78 plies! and the game still was equal...
(the evals also look so similarily in the beginning, that it also could be the user played the same NN for both sides?)

This is from the 'Daniel Rocha' games file

[pgn][Event "31970 vs 11248"] [Site "?"] [Date "2018.12.12"] [Round "1"] [White "31970"] [Black "11248"] [Result "1-0"] [ECO "C16"] [Opening "French"] [PlyCount "135"] [TimeControl "inf"] [Variation "Winawer, Advance Variation"] 1. e4 {book} e6 {book} 2. d4 {book} d5 {book} 3. Nc3 {book} Bb4 {book} 4. e5 {book} Ne7 {-0.64/9 1.7s} 5. a3 {+0.80/10 1.2s} Bxc3+ {-0.70/12 0.27s} 6. bxc3 {+0.75/11 0.20s} c5 {-0.61/11 0.10s} 7. Qg4 {+0.74/10 0.26s} Qc7 {-0.58/11 0.36s} 8. Qxg7 {+0.71/11 0.28s} Rg8 {-0.58/11 0.17s} 9. Qxh7 {+0.68/10 0.10s} cxd4 {-0.52/10 0.10s} 10. Ne2 {+0.63/9 0.10s} Nbc6 {-0.53/8 0.28s} 11. f4 {+0.66/8 0.20s} Bd7 {-0.53/8 0.58s} 12. Qd3 {+0.63/8 0.59s} dxc3 {-0.44/8 0.41s} 13. Qxc3 {+0.57/7 0.31s} O-O-O {-0.40/7 0.39s} 14. Ng3 {+0.59/7 0.58s} Nf5 {-0.47/8 0.70s} 15. Nxf5 {+0.73/8 0.65s} exf5 {-0.69/8 0.29s} 16. g3 {+0.92/8 0.20s} d4 {-1.10/7 0.72s} 17. Qc5 {+1.41/8 0.75s} b6 {-1.36/8 0.87s} 18. Qd6 {+1.50/9 0.91s} Qb7 {-0.97/9 1.5s} 19. Bg2 {+1.31/9 1.1s} Be8 {-1.17/10 0.30s} 20. Qf6 {+1.49/10 0.27s} Qd7 {-1.14/9 0.19s} 21. O-O {+1.16/11 0.42s} Rg6 {-0.45/11 0.61s} 22. Bxc6 {+0.51/10 0.64s} Qxc6 {-0.38/11 0.30s} 23. Qxc6+ {+0.62/9 0.60s} Rxc6 {-0.57/9 0.89s} 24. Rf2 {+0.68/8 0.34s} Rc5 {-0.47/8 0.58s} 25. Bb2 {+0.74/8 0.65s} Rb5 {-0.56/8 0.79s} 26. c4 {+0.83/8 0.60s} dxc3 {-0.76/8 0.84s} 27. Bxc3 {+0.92/7 0.48s} Bc6 {-0.95/7 0.73s} 28. Rc1 {+1.70/8 0.77s} Kb7 {-1.54/7 1.0s} 29. Bb4 {+1.83/8 0.40s} a5 {-1.53/8 0.45s} 30. Bd6 {+1.71/8 0.40s} Rh8 {-1.46/8 0.51s} 31. Rfc2 {+1.82/8 0.67s} Bd5 {-1.46/9 1.1s} 32. h4 {+1.62/8 0.69s} Rg8 {-1.20/9 0.60s} 33. Kf2 {+1.21/9 0.79s} Be4 {-0.70/10 0.70s} 34. Re2 {+0.79/10 0.80s} Rb3 {-0.51/10 0.40s} 35. Re3 {+0.65/10 0.27s} Rb2+ {-0.50/9 0.20s} 36. Re2 {+0.48/9 0.29s} Rb3 {-0.32/9 0.19s} 37. Re3 {+0.44/9 0.41s} Rb2+ {-0.31/7 0.28s} 38. Ke1 {+0.41/7 0.30s} Ka6 {-0.19/7 0.69s} 39. Rc7 {+0.33/7 0.67s} Bd5 {-0.13/7 0.50s} 40. Rcc3 {+0.30/7 0.50s} Rh2 {-0.12/7 0.70s} 41. Be7 {+0.32/7 0.91s} Be6 {-0.24/6 1.1s} 42. Red3 {+0.71/6 0.80s} Rh1+ {-0.67/6 1.2s} 43. Kf2 {+1.36/7 0.88s} Rh2+ {-1.07/6 1.1s} 44. Kg1 {+1.41/7 0.77s} Ra2 {-1.41/7 1.1s} 45. Rd8 {+1.71/8 0.50s} Rg7 {-1.63/7 0.66s} 46. Bg5 {+2.37/7 0.67s} Rh7 {-2.25/7 0.70s} 47. Rd6 {+2.77/7 0.49s} Rb2 {-2.42/7 0.60s} 48. Rcc6 {+2.83/7 0.80s} Rh8 {-2.78/8 0.81s} 49. Bd8 {+3.81/10 0.66s} Rg8 {-3.37/10 0.50s} 50. Rxb6+ {+3.95/11 0.30s} Rxb6 {-3.79/10 0.20s} 51. Bxb6 {+4.33/9 0.20s} Rxg3+ {-4.12/8 0.20s} 52. Kh2 {+4.59/8 0.25s} Rf3 {-4.68/8 0.62s} 53. h5 {+5.84/8 0.61s} Kb5 {-4.70/8 0.70s} 54. a4+ {+8.08/8 0.94s} Kxa4 {-8.05/8 1.2s} 55. h6 {+10.48/7 0.56s} Rxf4 {-10.57/7 0.50s} 56. Rd4+ {+17.06/8 0.50s} Rxd4 {-14.62/7 0.50s} 57. Bxd4 {+17.68/7 0.20s} f4 {-16.49/6 0.41s} 58. h7 {+31.56/6 0.60s} Kb3 {-27.14/5 0.82s} 59. h8=Q {+66.41/5 1.1s} a4 {-56.52/4 1.2s} 60. Qb8+ {+65.40/5 1.1s} Kc4 {-49.32/5 1.1s} 61. Ba1 {+55.87/5 1.1s} Kd3 {-42.18/5 1.2s} 62. Qb4 {+56.82/5 1.0s} f3 {-46.30/4 1.2s} 63. Kg3 {+60.70/5 1.2s} Ke2 {-59.30/5 1.2s} 64. Qe4+ {+64.93/5 1.0s} Kd2 {-57.61/5 1.0s} 65. Qxa4 {+64.20/5 0.91s} Ke2 {-59.00/5 1.2s} 66. Qc2+ {+70.75/5 1.0s} Ke3 {-57.55/5 0.92s} 67. Qf2+ {+66.81/5 1.0s} Ke4 {-70.21/5 0.95s} 68. Qxf3# {+128.00/4 0.79s, White mates} 1-0[/pgn]
[pgn] [Event "31970 vs 11248"] [Site "?"] [Date "2018.12.12"] [Round "1"] [White "11248"] [Black "31970"] [Result "0-1"] [ECO "C16"] [Opening "French"] [PlyCount "146"] [TimeControl "inf"] [Variation "Winawer, Advance Variation"] 1. e4 {book} e6 {book} 2. d4 {book} d5 {book} 3. Nc3 {book} Bb4 {book} 4. e5 {book} Ne7 {-0.64/9 1.5s} 5. a3 {+0.80/10 1.3s} Bxc3+ {-0.71/11 0.29s} 6. bxc3 {+0.74/11 0.20s} c5 {-0.60/10 0.19s} 7. Qg4 {+0.74/10 0.20s} Qc7 {-0.60/11 0.41s} 8. Qxg7 {+0.72/11 0.30s} Rg8 {-0.60/11 0.10s} 9. Qxh7 {+0.68/11 0.20s} cxd4 {-0.53/10 0.16s} 10. Ne2 {+0.63/9 0.16s} Nbc6 {-0.53/8 0.29s} 11. f4 {+0.64/8 0.30s} Bd7 {-0.54/8 0.60s} 12. Qd3 {+0.62/8 0.67s} dxc3 {-0.45/8 0.41s} 13. Qxc3 {+0.57/7 0.30s} O-O-O {-0.41/7 0.50s} 14. Ng3 {+0.58/7 0.66s} Nf5 {-0.47/8 0.69s} 15. Nxf5 {+0.69/8 0.70s} exf5 {-0.68/8 0.36s} 16. g3 {+0.94/8 0.20s} d4 {-1.10/7 0.78s} 17. Qc5 {+1.41/7 0.85s} b6 {-1.31/8 1.0s} 18. Qd6 {+1.53/8 0.91s} Qb7 {-1.14/10 1.6s} 19. Bg2 {+1.25/9 1.4s} Be8 {-1.31/10 0.23s} 20. Qf6 {+1.26/11 0.33s} Qd7 {-1.15/10 0.29s} 21. O-O {+1.10/11 0.40s} Rg6 {-0.46/11 0.70s} 22. Bxc6 {+0.54/10 0.69s} Qxc6 {-0.37/10 0.30s} 23. Qxc6+ {+0.60/9 0.77s} Rxc6 {-0.56/9 0.91s} 24. Rf2 {+0.66/8 0.35s} Rc5 {-0.47/8 0.61s} 25. Bb2 {+0.72/8 0.78s} Rb5 {-0.55/8 0.86s} 26. c4 {+0.82/8 0.69s} dxc3 {-0.71/8 0.81s} 27. Bxc3 {+0.92/7 0.48s} Bc6 {-0.90/7 0.81s} 28. Bb4 {+1.47/7 0.84s} Kb7 {-1.44/7 0.85s} 29. Rc1 {+1.88/8 0.70s} a5 {-1.51/8 0.80s} 30. Bd6 {+1.75/8 0.50s} Rh8 {-1.34/8 0.50s} 31. Rfc2 {+1.77/8 0.70s} Bd5 {-1.55/8 1.0s} 32. h4 {+1.67/8 0.70s} Rg8 {-1.21/9 0.64s} 33. Kf2 {+1.14/9 0.91s} Be4 {-0.71/10 0.89s} 34. Re2 {+0.73/10 1.1s} Rb3 {-0.53/10 0.40s} 35. Re3 {+0.67/10 0.41s} Rb2+ {-0.52/9 0.20s} 36. Re2 {+0.48/9 0.35s} Rb3 {-0.33/9 0.25s} 37. Re3 {+0.45/9 0.38s} Rb2+ {-0.32/7 0.29s} 38. Ke1 {+0.45/7 0.39s} Bd5 {-0.17/7 0.85s} 39. Rcc3 {+0.33/7 0.94s} Be6 {-0.07/7 1.0s} 40. Re2 {+0.30/8 1.0s} Rb1+ {-0.16/9 0.60s} 41. Kf2 {+0.19/8 0.54s} Rh1 {-0.02/7 0.40s} 42. Be7 {+0.30/8 0.40s} Re8 {-0.23/7 0.99s} 43. Bf6 {+0.74/8 1.1s} b5 {-0.48/8 1.3s} 44. Re1 {+0.69/9 1.1s} Rxe1 {-0.13/11 0.98s} 45. Kxe1 {+0.27/10 0.79s} b4 {-0.01/9 0.13s} 46. axb4 {+0.11/10 0.17s} axb4 {0.00/8 0.20s} 47. Rc5 {-0.01/8 0.60s} b3 {+0.23/8 0.66s} 48. Kd2 {-0.11/8 1.2s} Rg8 {+0.24/9 0.76s} 49. Bg5 {-0.03/8 0.60s} Ra8 {+0.28/8 0.40s} 50. h5 {-0.21/8 0.58s} Ra2+ {+0.71/9 0.52s} 51. Kc1 {-1.33/9 1.2s} Rh2 {+1.86/10 0.90s} 52. Rb5+ {-2.21/9 1.0s} Kc6 {+2.55/8 0.98s} 53. Rb8 {-2.56/7 0.53s} Kc5 {+3.85/7 0.87s} 54. h6 {-4.06/7 1.1s} Kc4 {+4.43/7 1.00s} 55. Bf6 {-4.00/7 1.1s} Rxh6 {+5.36/7 0.86s} 56. Rh8 {-4.47/8 0.80s} Rg6 {+5.13/8 0.40s} 57. Bg5 {-4.50/8 0.37s} Kc3 {+5.51/7 0.49s} 58. Rb8 {-4.68/7 0.89s} Rg7 {+5.38/7 0.79s} 59. Rb6 {-4.78/7 0.74s} Rg8 {+5.52/7 1.3s} 60. Rc6+ {-4.34/7 0.68s} Kd4 {+4.41/8 0.73s} 61. Rd6+ {-3.79/7 0.70s} Kc3 {+3.72/8 0.49s} 62. Rc6+ {-3.11/8 0.39s} Bc4 {+4.07/7 1.0s} 63. Bf6 {-3.83/7 1.6s} Kd3 {+5.15/7 0.55s} 64. Bg5 {-4.77/6 0.91s} Ra8 {+5.66/6 1.1s} 65. Bf6 {-5.08/6 0.90s} Be6 {+6.35/6 0.60s} 66. Kb1 {-5.56/6 0.69s} Ra2 {+6.40/5 0.70s} 67. Be7 {-6.04/5 0.95s} Bd5 {+9.44/6 0.90s} 68. Rc8 {-8.91/6 1.3s} Be4 {+14.79/7 0.68s} 69. Kc1 {-10.79/7 0.80s} b2+ {+17.03/8 0.59s} 70. Kb1 {-15.76/8 0.48s} Kd2+ {+25.17/8 0.30s} 71. Kxa2 {-31.03/8 0.60s} b1=Q+ {+45.51/6 0.30s} 72. Ka3 {-47.70/5 0.10s} Bd5 {+90.87/5 0.37s} 73. Rd8 {-54.36/4 0.53s} Qb3# {+128.00/2 0.28s, Black mates} 0-1[/pgn]

[Guenther]

Well, well, that analysis shows that in reality NN ID 31970 played against itself

Hopefully no intentional fake, but just mixed up NNs (5000 nodes identical eval/depth)

[Event "31970 vs 11248"]
[Site "?"]
[Date "2018.12.12"]
[Round "1"]
[White "31970"]
[Black "11248"]
[Result "1-0"]
[ECO "C16"]
[Opening "French"]
[PlyCount "135"]
[TimeControl "inf"]
[Variation "Winawer, Advance Variation"]

1. e4 {book} e6 {book} 2. d4 {book} d5 {book} 3. Nc3 {book} Bb4 {book}
4. e5 {book} Ne7 {-0.64/9 1.7s}

[Event "31970 vs 11248"]
[Site "?"]
[Date "2018.12.12"]
[Round "1"]
[White "11248"]
[Black "31970"]
[Result "0-1"]
[ECO "C16"]
[Opening "French"]
[PlyCount "146"]
[TimeControl "inf"]
[Variation "Winawer, Advance Variation"]

1. e4 {book} e6 {book} 2. d4 {book} d5 {book} 3. Nc3 {book} Bb4 {book}
4. e5 {book} Ne7 {-0.64/9 1.5s}

Code: Select all

1.201: < Found pb network file: ./ID11248
1.794: < Creating backend [cudnn]...
...
27.706: > position startpos moves e2e4 e7e6 d2d4 d7d5 b1c3 f8b4 e4e5
27.706: > go infinite
...
63.212: < info depth 7 seldepth 18 time 35501 nodes 3297 score cp -34 hashfull 15 nps 92 tbhits 0 pv g8e7 a2a3 b4c3 b2c3 c7c5 d1g4 d8c7 g4g7 h8g8 g7h7 c5d4 g1e2 d4c3 h2h4 b8c6 h1h3 c7e5 c1f4
64.226: < info depth 8 seldepth 18 time 36501 nodes 3526 score cp -34 hashfull 16 nps 96 tbhits 0 pv g8e7 a2a3 b4c3 b2c3 c7c5 d1g4 d8c7 g4g7 h8g8 g7h7 c5d4 g1e2 d4c3 h2h4 b8c6 h1h3 c7e5 c1f4
69.296: < info depth 8 seldepth 18 time 41590 nodes 4037 score cp -36 hashfull 18 nps 97 tbhits 0 pv g8e7 a2a3 b4c3 b2c3 c7c5 d1g4 d8c7 g4g7 h8g8 g7h7 c5d4 g1e2 d4c3 h2h4 b8c6 h1h3 c7e5 c1f4
74.319: < info depth 8 seldepth 18 time 46601 nodes 4511 score cp -37 hashfull 20 nps 96 tbhits 0 pv g8e7 a2a3 b4c3 b2c3 c7c5 d1g4 d8c7 g4g7 h8g8 g7h7 c5d4 g1e2 d4c3 h2h4 b8c6 h1h3 c7e5 c1f4
77.689: < info depth 8 seldepth 19 time 49975 nodes 4955 score cp -36 hashfull 22 nps 99 tbhits 0 pv g8e7 a2a3 b4c3 b2c3 c7c5 d1g4 d8c7 g4g7 h8g8 g7h7 c5d4 g1e2 d4c3 h2h4 b8c6 h1h3 c7e5 c1f4
82.712: < info depth 8 seldepth 19 time 55000 nodes 5295 score cp -36 hashfull 23 nps 96 tbhits 0 pv g8e7 a2a3 b4c3 b2c3 c7c5 d1g4 d8c7 g4g7 h8g8 g7h7 c5d4 g1e2 d4c3 h2h4 b8c6 h1h3 c7e5 c1f4
85.270: < info depth 8 seldepth 20 time 57559 nodes 5807 score cp -36 hashfull 26 nps 100 tbhits 0 pv g8e7 a2a3 b4c3 b2c3 c7c5 d1g4 d8c7 g4g7 h8g8 g7h7 c5d4 g1e2 b8c6 c3d4 c6d4 a1a2 d4e2 f1e2 c7e5 c1b2
90.325: < info depth 8 seldepth 20 time 62600 nodes 6347 score cp -35 hashfull 28 nps 101 tbhits 0 pv g8e7 a2a3 b4c3 b2c3 c7c5 d1g4 d8c7 g4g7 h8g8 g7h7 c5d4 g1e2 b8c6 c3d4 c6d4 a1a2 d4e2 f1e2 c7e5 c1b2

Code: Select all

0.858: < Found pb network file: ./ID31970
1.435: < Creating backend [cudnn]...
...
37.518: > position startpos moves e2e4 e7e6 d2d4 d7d5 b1c3 f8b4 e4e5
37.518: > go infinite
...
72.010: < info depth 9 seldepth 24 time 34492 nodes 3629 score cp -65 hashfull 15 nps 105 tbhits 0 pv g8e7 a2a3 b4c3 b2c3 c7c5 d1g4 d8c7 g4g7 h8g8 g7h7 c5d4 g1e2 b8c6 f2f4 c8d7 h7d3 d4c3 d3c3 e8c8 c1e3 e7f5 e3f2 d5d4
77.111: < info depth 9 seldepth 24 time 39592 nodes 4174 score cp -65 hashfull 17 nps 105 tbhits 0 pv g8e7 a2a3 b4c3 b2c3 c7c5 d1g4 d8c7 g4g7 h8g8 g7h7 c5d4 g1e2 b8c6 f2f4 c8d7 h7d3 d4c3 d3c3 e8c8 c1e3 e7f5 e3f2 d5d4
82.212: < info depth 9 seldepth 24 time 44692 nodes 4722 score cp -65 hashfull 19 nps 105 tbhits 0 pv g8e7 a2a3 b4c3 b2c3 c7c5 d1g4 d8c7 g4g7 h8g8 g7h7 c5d4 g1e2 b8c6 f2f4 c8d7 h7d3 d4c3 d3c3 e8c8 c1e3 e7f5 e3f2 d5d4
84.755: < info depth 9 seldepth 25 time 47241 nodes 5067 score cp -64 hashfull 20 nps 107 tbhits 0 pv g8e7 a2a3 b4c3 b2c3 c7c5 d1g4 d8c7 g4g7 h8g8 g7h7 c5d4 g1e2 b8c6 f2f4 d4c3 h7d3 c8d7 d3c3 e8c8 c1e3 e7f5 e3f2 d5d4 c3c5
87.454: < info depth 10 seldepth 25 time 49931 nodes 5368 score cp -65 hashfull 22 nps 107 tbhits 0 pv g8e7 a2a3 b4c3 b2c3 c7c5 d1g4 d8c7 g4g7 h8g8 g7h7 c5d4 g1e2 b8c6 f2f4 d4c3 h7d3 c8d7 d3c3 e8c8 c1e3 e7f5 e3f2 d5d4 c3c5

[mehmet karaman]

You are right for first and second game. Their moves are same in the first 37 moves.

But in the other games ı can't see this similarity.

[Guenther]

You are right for first and second game. Their moves are same in the first 37 moves.

But in the other games ı can't see this similarity.

I can see it and can prove it and of course all games were played in one chunk and have round tags added so the user 'Daniel Rocha'
played a complete match NN 31970 vs. 31970.

[Event "31970 vs 11248"]
[Site "?"]
[Date "2018.12.12"]
[Round "2"]
[White "31970"]
[Black "11248"]
[Result "1/2-1/2"]
[ECO "E10"]
[Opening "Queen's pawn game"]
[PlyCount "67"]
[TimeControl "inf"]

1. d4 {book} Nf6 {book} 2. c4 {book} e6 {book} 3. Nf3 {book} c5 {book}
4. d5 {book} d6 {-0.83/8 1.4s}

[Event "31970 vs 11248"]
[Site "?"]
[Date "2018.12.12"]
[Round "2"]
[White "11248"]
[Black "31970"]
[Result "1-0"]
[ECO "E10"]
[Opening "Queen's pawn game"]
[PlyCount "301"]
[TimeControl "inf"]

1. d4 {book} Nf6 {book} 2. c4 {book} e6 {book} 3. Nf3 {book} c5 {book}
4. d5 {book} d6 {-0.83/8 1.4s}

[Laskos]

For now 31970 is the best

31970- 11248: +9 -5 =186
Hardware used: 2080Ti and 6men TBs
Time control: 2min + 1sec
Book: Private (about 30 = positions)

31970- 11248: +39 - 27 =139
GPU 2080Ti and 6men TBs
5000 nodes/move

https://groups.google.com/forum/#!forum/lczero

Have they been tried aganst regular engines? Last time I checked, 2 days ago, test30 was still some 40 Elo points behind test10 and advancing very, very slowly, on the average of less than 0.2 Elo points per net on 700 nets span. Against SF8.
Also, don't use fixed nodes when testing strength.

[Guenther]

For now 31970 is the best

31970- 11248: +9 -5 =186
Hardware used: 2080Ti and 6men TBs
Time control: 2min + 1sec
Book: Private (about 30 = positions)

31970- 11248: +39 - 27 =139
GPU 2080Ti and 6men TBs
5000 nodes/move

https://groups.google.com/forum/#!forum/lczero

Have they been tried aganst regular engines? Last time I checked, 2 days ago, test30 was still some 40 Elo points behind test10 and advancing very, very slowly, on the average of less than 0.2 Elo points per net on 700 nets span. Against SF8.
Also, don't use fixed nodes when testing strength.

It seems you did not read the thread? Test B is discredited - games for test A are not available...