LCZero: Progress and Scaling. Relation to CCRL Elo

[CMCanavessi]

I m sorry, maybe I don't understand it well, but when I look this graph :
http://lczero.org/ I don't see progress in elo strenght since many games and days. So perhaps it is not the right thing to watch ?

Yes, it's been said several times that the graph is currently completely useless, and even broken due to the way it's calculated.

[JJJ]

So the only way to know the strenght is to test is ourself ?

[Guenther]

So the only way to know the strenght is to test is ourself ?

Yes, or trust people who already do this and post here for weeks ;-)

[JJJ]

So the only way to know the strenght is to test is ourself ?

Yes, or trust people who already do this and post here for weeks

all right thanks you !

[Guenther]

So the only way to know the strenght is to test is ourself ?

Yes, or trust people who already do this and post here for weeks ;-)

all right thanks you !

I want to add that LCZero is still quite early (around 1 month w/o too much bugs and enough capacity) in the training phase,
thus the developers and maintainers don't care much (yet) for stats,
which report 'ratings' for non training games, which can be compared to sth outside of the training process.
(Also it not as easy to standardize them, because of the more different hardware and tc used by testers.)
LCZero scales different than AB programs. It needs a certain min. number of playouts to compete and probably the strength grows non-linear.

[AdminX]

So the only way to know the strenght is to test is ourself ?

To show it's potential. Here is a game played versus Maverick 1.5 rated about 2500. (LCZero C183)

[pgn]
[Event "5 Minutes/Game"]
[Site "Engine Match"]
[Date "2018.04.12"]
[Round "2"]
[White "Maverick 1.5 x64"]
[Black "LCZero v0.5"]
[Result "1/2-1/2"]
[ECO "C11"]
[PlyCount "199"]
[EventDate "2018.??.??"]

1. e4 {[%eval 18,17] [%emt 0:00:08]} e6 {[%eval 4,19] [%emt 0:00:06] (e5)} 2.
d4 {[%eval 58,17] [%emt 0:00:08]} d5 {[%eval 5,19] [%emt 0:00:02]} 3. Nc3 {
[%eval 53,17] [%emt 0:00:05]} Nf6 {[%eval 3,19] [%emt 0:00:03] (Nc6)} 4. e5 {
[%eval 56,17] [%emt 0:00:07]} Nfd7 {[%eval -6,18] [%emt 0:00:03] (Ne4)} 5. Nf3
{[%eval 76,16] [%emt 0:00:05]} c5 {[%eval -10,18] [%emt 0:00:02]} 6. Nb5 {
[%eval 63,17] [%emt 0:00:10]} a6 {[%eval -19,19] [%emt 0:00:03] (cxd4)} 7. Nd6+
{[%eval 57,18] [%emt 0:00:05]} Bxd6 {[%eval -22,19] [%emt 0:00:01]} 8. exd6 {
[%eval 55,18] [%emt 0:00:05]} cxd4 {[%eval -21,19] [%emt 0:00:01] (Nc6)} 9.
Qxd4 {[%eval 85,18] [%emt 0:00:06]} O-O {[%eval -22,19] [%emt 0:00:03] (Nf6)}
10. Bg5 {[%eval 77,16] [%emt 0:00:05]} f6 {[%eval -26,19] [%emt 0:00:06]} 11.
Bd2 {[%eval 57,17] [%emt 0:00:04]} Nc6 {[%eval -31,18] [%emt 0:00:03]} 12. Qe3
{[%eval 24,20] [%emt 0:00:07]} Nde5 {[%eval -30,19] [%emt 0:00:05]} 13. Nxe5 {
[%eval 29,19] [%emt 0:00:04]} fxe5 {[%eval -24,19] [%emt 0:00:03]} 14. Qa3 {
[%eval 0,20] [%emt 0:00:27]} Qf6 {[%eval -55,19] [%emt 0:00:06] (Bd7)} 15. f3 {
[%eval 57,18] [%emt 0:00:08]} Qh4+ {[%eval -59,19] [%emt 0:00:04] (Rd8)} 16. g3
{[%eval 43,17] [%emt 0:00:04]} Qf6 {[%eval -63,19] [%emt 0:00:03]} 17. O-O-O {
[%eval 46,17] [%emt 0:00:04]} Qxf3 {[%eval -72,19] [%emt 0:00:04] (Bd7)} 18.
Qxf3 {[%eval 33,19] [%emt 0:00:03]} Rxf3 {[%eval -69,18] [%emt 0:00:01]} 19.
Bg2 {[%eval 25,19] [%emt 0:00:03]} Rf2 {[%eval -99,18] [%emt 0:00:03]} 20. Bh3
{[%eval 8,19] [%emt 0:00:05]} Bd7 {[%eval -104,18] [%emt 0:00:04]} 21. Be3 {
[%eval 2,19] [%emt 0:00:05]} Rf3 {[%eval -104,19] [%emt 0:00:05] (Rff8)} 22.
Bc5 {[%eval 22,20] [%emt 0:00:04]} Raf8 {[%eval -106,19] [%emt 0:00:05]} 23.
Bg2 {[%eval 11,21] [%emt 0:00:10]} R3f7 {[%eval -89,19] [%emt 0:00:06] (R3f6)}
24. Rhf1 {[%eval 13,20] [%emt 0:00:05]} e4 {[%eval -106,19] [%emt 0:00:05]} 25.
Rxf7 {[%eval 4,19] [%emt 0:00:06]} Kxf7 {[%eval -84,19] [%emt 0:00:05]} 26.
Rf1+ {[%eval -4,17] [%emt 0:00:07]} Kg8 {[%eval -119,19] [%emt 0:00:06]} 27.
Rxf8+ {[%eval -10,17] [%emt 0:00:03]} Kxf8 {[%eval -124,16] [%emt 0:00:00]} 28.
h3 {[%eval -13,16] [%emt 0:00:03]} h5 {[%eval -138,18] [%emt 0:00:06] (Kf7)}
29. h4 {[%eval -2,15] [%emt 0:00:02]} Kf7 {[%eval -144,18] [%emt 0:00:05]} 30.
b3 {[%eval -35,16] [%emt 0:00:06]} Kf6 {[%eval -137,18] [%emt 0:00:05] (e5)}
31. Bf1 {[%eval -42,17] [%emt 0:00:02]} Ke5 {[%eval -146,18] [%emt 0:00:05]
(e5)} 32. Be2 {[%eval -94,17] [%emt 0:00:05]} g6 {[%eval -134,18] [%emt 0:00:
05]} 33. g4 {[%eval -175,18] [%emt 0:00:07]} hxg4 {[%eval -176,17] [%emt 0:00:
04]} 34. Bxg4 {[%eval -184,17] [%emt 0:00:01]} Kf4 {[%eval -173,18] [%emt 0:00:
05] (Nd4)} 35. Be2 {[%eval -57,17] [%emt 0:00:02]} Ne5 {[%eval -157,19] [%emt
0:00:05] (d4)} 36. h5 {[%eval -42,17] [%emt 0:00:01]} gxh5 {[%eval -192,17]
[%emt 0:00:03]} 37. Bxh5 {[%eval -83,17] [%emt 0:00:01]} Nc6 {[%eval -186,19]
[%emt 0:00:05] (Nf3)} 38. Bg1 {[%eval -67,16] [%emt 0:00:04]} e5 {[%eval -202,
18] [%emt 0:00:06] (e3)} 39. Bf7 {[%eval -49,16] [%emt 0:00:01]} d4 {[%eval
-185,17] [%emt 0:00:02]} 40. Bd5 {[%eval -62,16] [%emt 0:00:03]} e3 {[%eval
-174,18] [%emt 0:00:03] (Kf5)} 41. Bh2+ {[%eval -103,17] [%emt 0:00:03]} Kf5 {
[%eval -158,17] [%emt 0:00:02]} 42. Bg8 {[%eval -164,18] [%emt 0:00:08]} Ke4 {
[%eval -197,17] [%emt 0:00:03] (b5)} 43. Bg3 {[%eval -128,17] [%emt 0:00:01]}
Kf3 {[%eval -219,17] [%emt 0:00:03] (Na7)} 44. Bh4 {[%eval -177,17] [%emt 0:00:
01]} Kg4 {[%eval -202,18] [%emt 0:00:04] (e4)} 45. Bf6 {[%eval -200,16] [%emt
0:00:01]} Kf5 {[%eval -204,18] [%emt 0:00:03] (Nb4)} 46. Bg7 {[%eval -111,15]
[%emt 0:00:01]} Ke4 {[%eval -192,18] [%emt 0:00:04] (b5)} 47. a4 {[%eval -74,
16] [%emt 0:00:01]} a5 {[%eval -195,18] [%emt 0:00:04] (Nd8)} 48. Bc4 {[%eval
-57,16] [%emt 0:00:01]} Bg4 {[%eval -171,18] [%emt 0:00:04] (Kf5)} 49. Bf6 {
[%eval -59,16] [%emt 0:00:01]} Bd7 {[%eval -175,17] [%emt 0:00:03] (Nb4)} 50.
Bg7 {[%eval 0,17] [%emt 0:00:01]} Kf4 {[%eval -167,18] [%emt 0:00:03]} 51. Bh6+
{[%eval -60,15] [%emt 0:00:02]} Kf3 {[%eval -166,18] [%emt 0:00:03] (Kf5)} 52.
Bg5 {[%eval -120,15] [%emt 0:00:01]} Kf2 {[%eval -183,17] [%emt 0:00:03] (e4)}
53. Bh4+ {[%eval -6,15] [%emt 0:00:00]} Kf3 {[%eval -172,17] [%emt 0:00:02]
(Kg1)} 54. Bg5 {[%eval 0,18] [%emt 0:00:01]} Kg3 {[%eval -174,18] [%emt 0:00:
03]} 55. Bd5 {[%eval -24,15] [%emt 0:00:01]} Kf2 {[%eval -189,17] [%emt 0:00:
02] (b6)} 56. Bh4+ {[%eval -14,17] [%emt 0:00:01]} Kf1 {[%eval -145,16] [%emt
0:00:02]} 57. Bc4+ {[%eval -38,17] [%emt 0:00:02]} Kg2 {[%eval -153,17] [%emt
0:00:01] (Kg1)} 58. Bd5+ {[%eval -13,17] [%emt 0:00:01]} Kf1 {[%eval -136,17]
[%emt 0:00:02] (Kh3)} 59. Bc4+ {[%eval 0,19] [%emt 0:00:01]} Kg2 {[%eval -137,
17] [%emt 0:00:01]} 60. Bd5+ {[%eval 0,19] [%emt 0:00:01]} Kg1 {[%eval -124,17]
[%emt 0:00:01]} 61. Be4 {[%eval -38,16] [%emt 0:00:00]} Kf1 {[%eval -131,17]
[%emt 0:00:02] (Nb4)} 62. Bd3+ {[%eval 0,17] [%emt 0:00:00]} Kg2 {[%eval -133,
16] [%emt 0:00:01]} 63. Be4+ {[%eval 0,18] [%emt 0:00:01]} Kf1 {[%eval -127,17]
[%emt 0:00:02] (Kg1)} 64. Bd3+ {[%eval 0,19] [%emt 0:00:01]} Kg2 {[%eval -122,
17] [%emt 0:00:01]} 65. Be4+ {[%eval 0,19] [%emt 0:00:00]} Kh3 {[%eval -116,17]
[%emt 0:00:01]} 66. Bf6 {[%eval 25,16] [%emt 0:00:00]} Kg4 {[%eval -123,16]
[%emt 0:00:01]} 67. Bd3 {[%eval -56,16] [%emt 0:00:01]} Kf4 {[%eval -140,17]
[%emt 0:00:02]} 68. Bc4 {[%eval -71,15] [%emt 0:00:00]} Bf5 {[%eval -158,17]
[%emt 0:00:02] (Nb4)} 69. Bb5 {[%eval -41,14] [%emt 0:00:00]} Bg4 {[%eval -144,
16] [%emt 0:00:01] (Kf3)} 70. Bg7 {[%eval -76,16] [%emt 0:00:02]} Bd7 {[%eval
-132,16] [%emt 0:00:01] (Be6)} 71. Bh6+ {[%eval -67,14] [%emt 0:00:00]} Kf3 {
[%eval -135,16] [%emt 0:00:01] (Kf5)} 72. Bg5 {[%eval -76,16] [%emt 0:00:00]}
Kf2 {[%eval -157,16] [%emt 0:00:01] (Be6)} 73. Bh4+ {[%eval -32,15] [%emt 0:00:
00]} Kf3 {[%eval -137,16] [%emt 0:00:01] (Kg2)} 74. Bg5 {[%eval 0,18] [%emt 0:
00:00]} e4 {[%eval -138,16] [%emt 0:00:01]} 75. Bf6 {[%eval -121,15] [%emt 0:
00:00]} e2 {[%eval -145,15] [%emt 0:00:01]} 76. Kd2 {[%eval -163,15] [%emt 0:
00:00]} d3 {[%eval -121,15] [%emt 0:00:01]} 77. Bxc6 {[%eval -179,15] [%emt 0:
00:00]} dxc2 {[%eval -129,16] [%emt 0:00:01] (Bxc6)} 78. Bh4 {[%eval -193,16]
[%emt 0:00:00]} Bxc6 {[%eval -183,15] [%emt 0:00:01]} 79. Kxc2 {[%eval -220,16]
[%emt 0:00:00]} e3 {[%eval -169,15] [%emt 0:00:01] (Kg2)} 80. Be1 {[%eval -79,
16] [%emt 0:00:00]} b6 {[%eval -159,15] [%emt 0:00:01] (Ke4)} 81. Kd3 {[%eval
-73,17] [%emt 0:00:00]} Bd7 {[%eval -148,15] [%emt 0:00:01] (Be8)} 82. Bh4 {
[%eval -68,17] [%emt 0:00:00]} b5 {[%eval -137,15] [%emt 0:00:01] (Be6)} 83.
axb5 {[%eval -79,17] [%emt 0:00:00]} Bxb5+ {[%eval -118,15] [%emt 0:00:01]} 84.
Kd4 {[%eval -74,19] [%emt 0:00:00]} Bd7 {[%eval -116,16] [%emt 0:00:01] (a4)}
85. Be1 {[%eval -74,17] [%emt 0:00:00]} a4 {[%eval -101,16] [%emt 0:00:01]} 86.
bxa4 {[%eval -57,18] [%emt 0:00:00]} Bxa4 {[%eval -99,15] [%emt 0:00:01]} 87.
Kd5 {[%eval -68,18] [%emt 0:00:01]} Bd7 {[%eval -106,15] [%emt 0:00:01] (Kf4)}
88. Ke5 {[%eval -53,17] [%emt 0:00:00]} Bg4 {[%eval -112,15] [%emt 0:00:01]
(Bc6)} 89. Ba5 {[%eval -37,18] [%emt 0:00:00]} Kf2 {[%eval -135,14] [%emt 0:00:
00]} 90. Kf4 {[%eval -28,18] [%emt 0:00:00]} Bc8 {[%eval -146,15] [%emt 0:00:
00] (Bd7)} 91. Bd8 {[%eval -32,19] [%emt 0:00:01]} Kf1 {[%eval -150,14] [%emt
0:00:00] (e1Q)} 92. Ba5 {[%eval -7,16] [%emt 0:00:00]} Kf2 {[%eval -109,14]
[%emt 0:00:00] (Be6)} 93. Bd8 {[%eval 0,20] [%emt 0:00:00]} Kf1 {[%eval -132,
15] [%emt 0:00:00]} 94. Ba5 {[%eval 0,18] [%emt 0:00:00]} e1=Q {[%eval -18,14]
[%emt 0:00:00]} 95. Bxe1 {[%eval -8,17] [%emt 0:00:00]} e2 {[%eval -9,14]
[%emt 0:00:00]} 96. Bc3 {[%eval -8,17] [%emt 0:00:00]} Kf2 {[%eval -20,14]
[%emt 0:00:00]} 97. Bd4+ {[%eval -8,16] [%emt 0:00:00]} Kf1 {[%eval -40,14]
[%emt 0:00:00] (Ke1)} 98. Bc3 {[%eval 0,14] [%emt 0:00:00]} Kf2 {[%eval -16,15]
[%emt 0:00:00]} 99. Bd4+ {[%eval 0,18] [%emt 0:00:00]} Kf1 {[%eval -18,14]
[%emt 0:00:00]} 100. Bc3 {[%eval 0,13] [%emt 0:00:00]} 1/2-1/2

[/pgn]

[Martin Hertz]

So the only way to know the strenght is to test is ourself ?

M-Chess 5 (1995)
Real DOS @ 3.3 GHz
5 sec/move average
Book off
Ponder on

LCZero ID 122
Web
Hard

M-Chess with white:

1. e4 e6
2. d4 d5
3. exd5 exd5
4. Bd3 c5
5. Nf3 cxd4
6. Nxd4 Nf6
7. O-O Be7
8. Re1 Nc6
9. Be3 Ng4
10. Nxc6 bxc6
11. Bc5 Be6
12. Rxe6 fxe6
13. Qxg4 Bxc5
14. Qxe6+ Kf8
15. Qxc6 Rc8
16. Qe6 g6
17. Nc3 Kg7
18. Qe5+ Qf6
19. Qxf6+ Kxf6
20. Nxd5+ Ke5
21. Nc3 Rhe8
22. Re1+ Kd6
23. Rxe8 Rxe8
24. Ne4+ Kc6
25. Kf1 Bb6
26. b4 Bc7
27. g3 Re5
28. c4 g5
29. Nf6 Bd8
30. Nxh7 a5
31. b5+ Kd6
32. Nf8 Re8
33. Ng6 Kc5
34. h4 gxh4
35. gxh4 Kd4
36. Be2 Rg8
37. h5 Rg7
38. Nf8 Ke5
39. f4+ Kf6
40. c5 Ke7
41. Ng6+ Kf6
42. Ne5 Ke6
43. Bd3 Rg3
44. h6 Bf6
45. Bc4+ Kf5
46. b6 Kxf4
47. Nd3+ Kf3
48. b7 Rh3
49. Ne1+ Ke4
50. b8=Q Be5
51. Qb7+ Ke3
52. Qb3+ Kd4
53. Nc2+ Kxc5
54. Qb5+ Kd6
55. Qd5+ Kc7
56. Qxe5+ Kc8
57. Be6+ Kb7
58. Qb5+ Kc7
59. Qc5+ Kb7
60. Bd5+ Kb8
61. Qb6+ Kc8
62. Be6#

M-Chess with black:

1. e4 e5
2. Nf3 Nf6
3. Nc3 Nc6
4. Bb5 Bb4
5. Bxc6 dxc6
6. d3 Bg4
7. h3 Bxf3
8. Qxf3 Qd4
9. Bd2 O-O
10. O-O-O a5
11. Qg3 Rfd8
12. a3 Bc5
13. f4 Nh5
14. Qf3 Nxf4
15. Bxf4 exf4
16. Qxf4 Bxa3
17. Rdf1 f6
18. Rf3 Qxc3
19. bxa3 Qa1+
20. Kd2 Qxh1
21. Rg3 Rd6
22. Qg4 g6
23. h4 Qa1
24. h5 g5
25. Rf3 Re8
26. h6 Qxa3
27. Qf5 Qb4+
28. Ke2 Qc3
29. Kf2 a4
30. g3 a3
31. Kg2 a2
32. Rf1 Qxc2+
33. Rf2 Qxf2+
34. Kxf2 a1=Q
35. Kg2 Rxd3
36. e5 Rd2+
37. Kh3 Qd1
38. Qxf6 Qh5#

[AdminX]

[pgn][Event "Skittle"]
[Site "Play.LCZero.Org"]
[Date "2018.04.12"]
[Round "?"]
[White "MChess, 5."]
[Black "LCZero, ID 122 Hard"]
[Result "1-0"]
[PlyCount "123"]
[EventDate "2018.04.12"]

1. e4 e6 2. d4 d5 3. exd5 exd5 4. Bd3 c5 5. Nf3 cxd4 6. Nxd4 Nf6 7. O-O Be7 8.
Re1 Nc6 9. Be3 Ng4 10. Nxc6 bxc6 11. Bc5 Be6 12. Rxe6 fxe6 13. Qxg4 Bxc5 14.
Qxe6+ Kf8 15. Qxc6 Rc8 16. Qe6 g6 17. Nc3 Kg7 18. Qe5+ Qf6 19. Qxf6+ Kxf6 20.
Nxd5+ Ke5 21. Nc3 Rhe8 22. Re1+ Kd6 23. Rxe8 Rxe8 24. Ne4+ Kc6 25. Kf1 Bb6 26.
b4 Bc7 27. g3 Re5 28. c4 g5 29. Nf6 Bd8 30. Nxh7 a5 31. b5+ Kd6 32. Nf8 Re8 33.
Ng6 Kc5 34. h4 gxh4 35. gxh4 Kd4 36. Be2 Rg8 37. h5 Rg7 38. Nf8 Ke5 39. f4+ Kf6
40. c5 Ke7 41. Ng6+ Kf6 42. Ne5 Ke6 43. Bd3 Rg3 44. h6 Bf6 45. Bc4+ Kf5 46. b6
Kxf4 47. Nd3+ Kf3 48. b7 Rh3 49. Ne1+ Ke4 50. b8=Q Be5 51. Qb7+ Ke3 52. Qb3+
Kd4 53. Nc2+ Kxc5 54. Qb5+ Kd6 55. Qd5+ Kc7 56. Qxe5+ Kc8 57. Be6+ Kb7 58. Qb5+
Kc7 59. Qc5+ Kb7 60. Bd5+ Kb8 61. Qb6+ Kc8 62. Be6# 1-0

[/pgn]

[pgn][Event "Skittle"]
[Site "Play.LCZero.Org"]
[Date "2018.04.12"]
[Round "?"]
[White "LCZero, ID 122 Hard"]
[Black "MChess, 5."]
[Result "0-1"]
[PlyCount "76"]
[EventDate "2018.04.12"]

1. e4 e5 2. Nf3 Nf6 3. Nc3 Nc6 4. Bb5 Bb4 5. Bxc6 dxc6 6. d3 Bg4 7. h3 Bxf3 8.
Qxf3 Qd4 9. Bd2 O-O 10. O-O-O a5 11. Qg3 Rfd8 12. a3 Bc5 13. f4 Nh5 14. Qf3
Nxf4 15. Bxf4 exf4 16. Qxf4 Bxa3 17. Rdf1 f6 18. Rf3 Qxc3 19. bxa3 Qa1+ 20. Kd2
Qxh1 21. Rg3 Rd6 22. Qg4 g6 23. h4 Qa1 24. h5 g5 25. Rf3 Re8 26. h6 Qxa3 27.
Qf5 Qb4+ 28. Ke2 Qc3 29. Kf2 a4 30. g3 a3 31. Kg2 a2 32. Rf1 Qxc2+ 33. Rf2
Qxf2+ 34. Kxf2 a1=Q 35. Kg2 Rxd3 36. e5 Rd2+ 37. Kh3 Qd1 38. Qxf6 Qh5# 0-1

[/pgn]

[Werewolf]

No progress? We have just tested a new bigger net (128x10) and it's 200+ elo stronger. Also v0.5 has been released which brings a huge boost in strenght to the current (and bugged) networks.

Right now the training pipeline is stopped while bugs are fixed and once everything is stable enough, it will continue, probably with the new bigger net.

I'll have to change my gauntlet engines because they are already too weak and "BigNet" (which we call the new enlarged net) crushed like they were toys.

128x10 will probably be retrained, with an improved training schedule. We can expect another strength boost from that, compared to "BigNet" (whoever came up with that name...)

In a few days, we should have a Leela that can compete in the 2300-2500 range...

On my machine (1060 GPU so not amazingly fast) 107 + 0.5 network is consistently beating dedicated machines in the 2000 - 2300 range.

[CMCanavessi]

Well, finally the 128 network has been made public. I'm broadcasting right now a 10-game match between LCZero ID 123 Vs. Fruit 1.0

Time control is 40/40, 5-move book with reverse games and Fruit has ponder on (which LCZero doesn't support). Even with that advantage in favor of Fruit, Leela is winning 1-0 and about to win the reverse game also, which would be 2-0.

Fruit 1.0 is 2400 elo btw.

You can follow the games live at https://www.twitch.tv/ccls