LCZero: Progress and Scaling. Relation to CCRL Elo

[Laskos]

Well, finally the 128 network has been made public. I'm broadcasting right now a 10-game match between LCZero ID 123 Vs. Fruit 1.0

Time control is 40/40, 5-move book with reverse games and Fruit has ponder on (which LCZero doesn't support). Even with that advantage in favor of Fruit, Leela is winning 1-0 and about to win the reverse game also, which would be 2-0.

Fruit 1.0 is 2400 elo btw.

You can follow the games live at

You look like an insider there, so you might know about some upcoming changes. Do you know when the the v0.5 client will be updated? It seems buggy is some respects, for example "go infinite" doesn't work, which impedes me to test on test-suites in Polyglot.

Then, as my GT 730 GPU is an utter crap, with the new 128x10 networks CPU speed gets hit hard. A factor of 2 or so at its peak on initial positions (from 2000 NPS to 1000 or so). But even worse at short time controls I was testing before, maybe a factor of 4-5. Look at this progression in NPS with the new net:

Code: Select all

FEN: rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1

LC0 CPU 4 threads ID123:

 d      t      n      nps   
=============================================================
 6	00:00	 2		37	-2.21	e2-e4 c7-c5
 7	00:00	 3		50	-1.84	e2-e4 c7-c5 c2-c3
 9	00:00	 8		103	-1.07	e2-e4 c7-c5 c2-c3 e7-e6 d2-d4
 10	00:00	 14		138	-0.73	e2-e4 c7-c5 c2-c3 e7-e6 d2-d4
 11	00:00	 25		185	-0.49	e2-e4 c7-c5 c2-c3 e7-e6 d2-d4 d7-d5
 12	00:00	 46		216	-0.27	e2-e4 e7-e6 d2-d4 d7-d5 e4-e5 c7-c5 c2-c3
 13	00:00	 80		239	-0.15	e2-e4 c7-c5 c2-c3 e7-e6 d2-d4 d7-d5 e4xd5
 14	00:00	 141	253	-0.07	e2-e4 e7-e6 c2-c4 c7-c5 Ng1-f3 Nb8-c6 Nb1-c3 Ng8-f6 Bf1-e2 e6-e5 d2-d3
 15	00:00	 244	274	-0.01	e2-e4 e7-e6 c2-c4 c7-c5 Ng1-f3 Nb8-c6 Nb1-c3 Ng8-f6 Bf1-e2 e6-e5 d2-d3 d7-d6
 16	00:01	 423	305	+0.03	e2-e4 e7-e6 d2-d4 d7-d5 e4-e5 c7-c5 c2-c3 Nb8-c6 Ng1-f3 Ng8-e7 Bf1-d3 Ne7-g6 h2-h4 h7-h6 Bd3xg6
 17	00:02	 744	323	+0.06	e2-e4 e7-e6 d2-d4 d7-d5 e4-e5 c7-c5 c2-c3 Nb8-c6 Ng1-f3 Ng8-e7 Bf1-d3 Ne7-g6 h2-h4 h7-h6 Bd3xg6
 18	00:03	 1k		343	+0.07	e2-e4 c7-c5 Ng1-f3 e7-e6 d2-d4 c5xd4 Nf3xd4 Ng8-f6 Nb1-c3 Bf8-b4 e4-e5 Nf6-e4 Qd1-g4 Ne4xc3
 19	00:06	 2k		374	+0.09	e2-e4 e7-e6 d2-d4 d7-d5 e4-e5 c7-c5 c2-c3 c5xd4 c3xd4 Nb8-c6 Ng1-f3 Ng8-e7 Bf1-d3 Nc6-b4 Bd3-e2
 20	00:10	 4k		408	+0.10	e2-e4 e7-e6 d2-d4 d7-d5 e4-e5 c7-c5 c2-c3 c5xd4 c3xd4 Bf8-b4+ Nb1-c3 Ng8-e7 a2-a3 Bb4xc3+ b2xc3 Qd8-a5
 21	00:16	 8k		450	+0.10	e2-e4 e7-e6 c2-c4 c7-c5 Ng1-f3 Nb8-c6 Nb1-c3 Ng8-f6 Bf1-e2 e6-e5 d2-d3 d7-d6 h2-h3 Bf8-e7 a2-a3
 22	00:26	 13k	515	+0.10	e2-e3 Ng8-f6 d2-d4 e7-e6 c2-c4 d7-d5 Ng1-f3 c7-c5 c4xd5 e6xd5 Bf1-b5+ Nb8-c6 O-O Bf8-e7 d4xc5 Be7xc5 Nb1-c3
 23	00:41	 24k	586	+0.09	e2-e3 Ng8-f6 d2-d4 e7-e6 c2-c4 d7-d5 Ng1-f3 c7-c5 c4xd5 e6xd5 Bf1-b5+ Nb8-c6 O-O Bf8-e7 d4xc5 Be7xc5 Nb1-c3
 24	01:05	 43k	655	+0.09	e2-e3 Ng8-f6 c2-c4 e7-e5 d2-d4 e5xd4 e3xd4 Bf8-b4+ Nb1-c3 d7-d5 Ng1-f3 O-O Bf1-e2 d5xc4 Be2xc4 Rf8-e8+ Bc1-e3 Nf6-g4 O-O Ng4xe3 f2xe3 Re8xe3 Qd1-b3
 25	01:39	 76k	765	+0.09	e2-e3 Ng8-f6 c2-c4 e7-e5 d2-d4 e5xd4 e3xd4 Bf8-b4+ Nb1-c3 d7-d5 Ng1-f3 O-O Bf1-e2 d5xc4 Be2xc4 c7-c5 O-O c5xd4 Nf3xd4 Nb8-c6 Nd4xc6 b7xc6 Qd1xd8 Rf8xd8
 26	02:42	 135k	832	+0.09	e2-e3 Ng8-f6 c2-c4 e7-e5 d2-d4 e5xd4 e3xd4 Bf8-b4+ Nb1-c3 d7-d5 Ng1-f3 O-O Bf1-e2 d5xc4 Be2xc4 c7-c5 O-O c5xd4 Nf3xd4 Nb8-c6 Nd4xc6 b7xc6 Qd1xd8 Rf8xd8
 27	04:10	 241k	961	+0.09	c2-c4 Ng8-f6 Ng1-f3 e7-e6 e2-e3 d7-d5 d2-d4 c7-c5 c4xd5 e6xd5 Bf1-b5+ Nb8-c6 O-O Bf8-e7 d4xc5 Be7xc5 Nb1-c3 O-O h2-h3
 28	07:39	 426k	927	+0.08	c2-c4 Ng8-f6 e2-e3 c7-c5 Nb1-c3 e7-e6 Ng1-f3 d7-d5 c4xd5 e6xd5 d2-d4 c5xd4 e3xd4 Bf8-d6 Bf1-d3 O-O O-O Nb8-c6 Bc1-g5 h7-h6 Bg5xf6 Qd8xf6
 29	13:50	 748k	900	+0.07	c2-c4 Ng8-f6 Ng1-f3 c7-c5 Nb1-c3 Nb8-c6 e2-e3 e7-e6 d2-d4 d7-d5 c4xd5 e6xd5 Bf1-e2 c5xd4 e3xd4 Bf8-e7 O-O O-O h2-h3 h7-h6 a2-a3 a7-a6 Bc1-e3

NPS maxes at depth 27 after a whopping 4 minutes on 4 threads (i7 Haswell 3.8 GHz). And the increase in NPS is 3-fold from 1-3 seconds per move to 4 minutes per move. I will have some difficulty checking for "absolute" (say CCRL) strength without a very strong GPU, I hope the progress will be visible even in short tc games on CPU, although "absolute" ratings will be way off (by hundreds of Elo points).

[Laskos]

NPS maxes at depth 27 after a whopping 4 minutes on 4 threads (i7 Haswell 3.8 GHz). And the increase in NPS is 3-fold from 1-3 seconds per move to 4 minutes per move. I will have some difficulty checking for "absolute" (say CCRL) strength without a very strong GPU, I hope the progress will be visible even in short tc games on CPU, although "absolute" ratings will be way off (by hundreds of Elo points).

I am still tracking the progress with my meager CPU means. At 1s/move, I took Jabba 1.0 (about 2050 Elo CCRL) reference in matches of 200 games. First and second "bignets" performed as:

ID123: 57.5/200
ID124: 65.0/200

Although still within error margins, it seems it progresses fast, about +30 Elo points from one net to the successive one. I will maybe do a larger test when more progress accumulates (after several nets).

[AdminX]

Saw this today.

[JJJ]

I don't know about the human FIDE estimate, but it would be nice to see some official matches at each version to check !

[Werewolf]

NPS maxes at depth 27 after a whopping 4 minutes on 4 threads (i7 Haswell 3.8 GHz). And the increase in NPS is 3-fold from 1-3 seconds per move to 4 minutes per move. I will have some difficulty checking for "absolute" (say CCRL) strength without a very strong GPU, I hope the progress will be visible even in short tc games on CPU, although "absolute" ratings will be way off (by hundreds of Elo points).

I am still tracking the progress with my meager CPU means. At 1s/move, I took Jabba 1.0 (about 2050 Elo CCRL) reference in matches of 200 games. First and second "bignets" performed as:

ID123: 57.5/200
ID124: 65.0/200

Although still within error margins, it seems it progresses fast, about +30 Elo points from one net to the successive one. I will maybe do a larger test when more progress accumulates (after several nets).

I'm currently testing against Colossus 2008b - CCRL 2642

I'm using Noomen short and 15 seconds per move each.

Currently LCZero 127 is leading...

[Werewolf]

Very early days, but current result for LCZero 127 on NVidia 1060 vs Colossus 2008b @ 15 sec / move below. Colossus is single Intel Broadwell core @ 4.2 GHz.

4 wins
2 losses
1 draw

for LCZero

[hgm]

No progress? We have just tested a new bigger net (128x10) and it's 200+ elo stronger.

What kind of filters are these networks using? Is that 3x3 convolutions like AlphaZero? These might be very good for Go, but for Chess, where there are sliders, they are blind to a lot of elementary patterns, such as pins and discovered threats. They would have to make up for that by cascading many blocks to even get the whole significant part 'in view' (e.g. a Rook on e1 pinning a Queen on e4 against a King on e8).

I wonder what would happen if you used filters that each examined one file, one rank and one diagonal as well. These would only look at 8 squares, instead of 9, so it is not more complex. E.g. of every 8 filters you could put 1 on each ray, and leave 4 to examine the local 3x3 neighborhood. That might give you a much more powerful network for the same number of weights.

[JJJ]

I managed to have a boring draw. I think it is easier to have a draw than a win if you play safe move.
[pgn]
1. d4 Nf6
2. c4 e6
3. Nf3 b6
4. Nc3 Bb7
5. e3 g6
6. Be2 d6
7. O-O Bg7
8. Qc2 c5
9. b3 Nc6
10. Bb2 cxd4
11. Nxd4 Nxd4
12. exd4 Rc8
13. Qd1 a5
14. Bf3 Bxf3
15. Qxf3 d5
16. cxd5 Nxd5
17. Nxd5 Qxd5
18. Qxd5 exd5
19. Rfe1+ Kd7
20. Rac1 Rcd8
21. Kf1 Bf6
22. Rc2 a4
23. b4 Rb8
24. Rec1 Rhc8
25. Rxc8 Rxc8
26. Rxc8 Kxc8
27. Ke2 b5
28. g3 Be7
29. a3 f6
30. Ke3 Kd7
31. f4 Ke6
32. g4 h6
33. h4 f5
34. g5 h5
35. Bc3 Bd6
36. Bb2 Kd7
37. Bc3 Ke6
38. Bd2 Ke7
39. Kd3 Kd7
40. Be3 Ke8
41. Bd2 Bb8
42. Be3 Bd6
43. Bd2 Ke7
44. Ke3 Ke6
45. Kd3 Bc7
46. Ke3 Kf7
47. Kd3 Ke6
48. Ke3 Kf7
49. Kd3 Bb8
50. Ke3 Bd6
51. Kd3 Ke6
52. Ke3 Ke7
53. Kd3 Ke8
54. Ke3 Bb8
55. Kd3 Kd7
56. Ke3 Kc8
57. Kd3 Kd7
58. Ke3 Kc8
59. Kd3 Bd6
60. Ke3 Kd7
61. Kd3 Kc6
62. Ke3 Kc7
63. Kd3 Kc8
64. Ke3 Kd7
65. Kd3 Bc7
66. Ke3 Bb6
67. Kd3 Ke6
68. Ke3 Ba7
69. Kd3 Bb6
70. Ke3 Ke7
71. Kd3 Ke8
72. Ke3 Bd8
73. Kd3 Bb6
74. Ke3 Kd7
75. Kd3 Kc6
76. Ke3 Kc7
77. Kd3 Kd7
78. Ke3 Bc7
79. Kd3 Kc8
80. Ke3 Kb7
81. Kd3 Kc8
82. Ke3 Kd7
83. Kd3 Bd6
84. Ke3 Ke6 [/pgn]

[Laskos]

Saw this today.

Thanks! Very strong computer, and LC0 benefits from that, I cannot achieve such high results. TC seems to be 1+1, right?

[Laskos]

Very early days, but current result for LCZero 127 on NVidia 1060 vs Colossus 2008b @ 15 sec / move below. Colossus is single Intel Broadwell core @ 4.2 GHz.

4 wins
2 losses
1 draw

for LCZero

Wow, on a good GPU and longer time control, LC0 rocks. It scales completely differently from standard engines, give it strong hardware and LTC, and it soares.