LCZero: Progress and Scaling. Relation to CCRL Elo

[jkiliani]

On tactical test suite, also no significant progress. Let's hope for now that move selection in the openings will change this real stalling (that in self-games was artificial). It surely was stuck with some local optima in the openings, and the whole training process was flawed. If not this is the issue, a larger block/filters network will be required.

The change with opening move selection is only going to affect matches, which should yield more accurate self-play Elo ratings in the future. It did not and will not affect the training process at all, since training games always had the full variety in move selection. The most likely candidate for the stall is really the network size, which will be increased within a few days if all goes as planned.

[Werewolf]

Changing to v0.5 must add elo even without changing weight version, since on my machine v0.5 is noticeably faster.

My Nvidia 1060 used to get 2000 (is it nps or rollouts per second??) ; now it gets 3000.

[Guenther]

Changing to v0.5 must add elo even without changing weight version, since on my machine v0.5 is noticeably faster.

My Nvidia 1060 used to get 2000 (is it nps or rollouts per second??) ; now it gets 3000.

I guess your numbers are the reported nps, but if you would enable a log or debug output you could see it directly.

On my very slow GT 710 the nps increased by 3-4 times.
Here is a tiny excerpt for a previous test match and new version with the same conditions.

There is one caveat though, this time I used the full-tuner option before, thus I cannot say (yet) which of both influenced how much.
e.g.

Code: Select all

lczero05ID107.exe --tune-only --full-tuner -w c7479f15

There is also a cosmetic uci output change and now 'info string' is correctly added for making all different node stats visible in uci GUIs/parser

Code: Select all

16461 >LCZero_04ID107(0): position startpos moves b2b3 e7e5 c1b2 b8c6 e2e3 d7d6 g1f3 g8f6 d2d3 f8e7 f1e2 e8g8
16462 >LCZero_04ID107(0): isready
16462 <LCZero_04ID107&#40;0&#41;&#58; Unknown command&#58; ucinewgame
16462 <LCZero_04ID107&#40;0&#41;&#58; readyok
16462 >LCZero_04ID107&#40;0&#41;&#58; go wtime 1200000 btime 150000 movestogo 34
16925 <LCZero_04ID107&#40;0&#41;&#58; info depth 7 nodes 3 nps 5 score cp -101 winrate 24.62% time 387 pv h2h3 h7h6
16948 <LCZero_04ID107&#40;0&#41;&#58; info depth 9 nodes 11 nps 20 score cp -43 winrate 38.27% time 494 pv b1d2 h7h6 h2h3
16971 <LCZero_04ID107&#40;0&#41;&#58; info depth 10 nodes 13 nps 23 score cp -38 winrate 39.65% time 518 pv h2h3 h7h6 b1d2 f8e8
17053 <LCZero_04ID107&#40;0&#41;&#58; info depth 11 nodes 21 nps 33 score cp -28 winrate 42.23% time 600 pv h2h3 h7h6 e1g1 f8e8
17208 <LCZero_04ID107&#40;0&#41;&#58; info depth 12 nodes 37 nps 48 score cp -20 winrate 44.49% time 755 pv b1d2 h7h6 h2h3 f8e8
17509 <LCZero_04ID107&#40;0&#41;&#58; info depth 13 nodes 68 nps 64 score cp -13 winrate 46.36% time 1054 pv h2h3 h7h6 e1g1 f8e8 b1d2
18000 <LCZero_04ID107&#40;0&#41;&#58; info depth 14 nodes 119 nps 76 score cp -9 winrate 47.50% time 1547 pv b1d2 h7h6 h2h3 f8e8 e1g1
18876 <LCZero_04ID107&#40;0&#41;&#58; info depth 15 nodes 214 nps 88 score cp -6 winrate 48.20% time 2423 pv b1d2 h7h6 h2h3 f8e8 e1g1 d6d5
20341 <LCZero_04ID107&#40;0&#41;&#58; info depth 16 nodes 381 nps 98 score cp -5 winrate 48.60% time 3888 pv b1d2 h7h6 h2h3 f8e8 e1g1 d6d5
22554 <LCZero_04ID107&#40;0&#41;&#58; info depth 17 nodes 681 nps 111 score cp -3 winrate 49.03% time 6101 pv b1d2 h7h6 h2h3 f8e8 e1g1 d6d5 f3e5
26865 <LCZero_04ID107&#40;0&#41;&#58; info depth 18 nodes 1221 nps 117 score cp -2 winrate 49.30% time 10411 pv b1d2 h7h6 h2h3 f8e8 e1g1 e7f8 a2a3
33878 <LCZero_04ID107&#40;0&#41;&#58; info depth 19 nodes 2194 nps 126 score cp -2 winrate 49.40% time 17425 pv b1d2 h7h6 h2h3 f8e8 e1g1 e7f8 a2a3 d6d5
44728 <LCZero_04ID107&#40;0&#41;&#58; info depth 20 nodes 3939 nps 139 score cp -1 winrate 49.57% time 28275 pv b1d2 h7h6 h2h3 f8e8 e1g1 e7f8 a2a3 d6d5
49855 <LCZero_04ID107&#40;0&#41;&#58; info depth 20 nodes 4799 nps 144 score cp -1 winrate 49.62% time 33402 pv b1d2 h7h6 h2h3 f8e8 e1g1 e7f8 a2a3 d6d5 b3b4
49856 <LCZero_04ID107&#40;0&#41;&#58; bestmove b1d2

Code: Select all

13932 >LCZero_05ID107(0): position startpos moves b2b3 e7e5 c1b2 b8c6 e2e3 d7d6 g1f3 g8f6 d2d3 f8e7 f1e2 e8g8
13933 >LCZero_05ID107(0): isready
13933 <LCZero_05ID107&#40;0&#41;&#58; readyok
13933 >LCZero_05ID107&#40;0&#41;&#58; go wtime 1200000 btime 150000 movestogo 34
14009 <LCZero_05ID107&#40;0&#41;&#58; info depth 7 nodes 2 nps 11 score cp -130 winrate 19.25% time 86 pv O-O h6
14025 <LCZero_05ID107&#40;0&#41;&#58; info depth 8 nodes 5 nps 38 score cp -77 winrate 29.96% time 103 pv h3 h6 Nbd2
14031 <LCZero_05ID107&#40;0&#41;&#58; info depth 9 nodes 7 nps 55 score cp -61 winrate 33.84% time 109 pv h3 h6 Nbd2 Re8
14052 <LCZero_05ID107&#40;0&#41;&#58; info depth 10 nodes 12 nps 84 score cp -42 winrate 38.50% time 130 pv O-O h6 h3 Re8 Nbd2 d5
14083 <LCZero_05ID107&#40;0&#41;&#58; info depth 11 nodes 22 nps 130 score cp -28 winrate 42.34% time 161 pv O-O h6 h3 Re8 Nbd2 d5
14131 <LCZero_05ID107&#40;0&#41;&#58; info depth 12 nodes 37 nps 171 score cp -19 winrate 44.59% time 209 pv h3 h6 Nbd2 Re8 O-O Bf8 a3
14209 <LCZero_05ID107&#40;0&#41;&#58; info depth 13 nodes 67 nps 229 score cp -13 winrate 46.25% time 287 pv h3 h6 O-O Re8 Nbd2 Bf8 a3
14361 <LCZero_05ID107&#40;0&#41;&#58; info depth 14 nodes 120 nps 270 score cp -9 winrate 47.36% time 439 pv Nbd2 h6 h3 Re8 O-O d5 Nxe5
14611 <LCZero_05ID107&#40;0&#41;&#58; info depth 15 nodes 212 nps 306 score cp -7 winrate 48.01% time 689 pv Nbd2 h6 h3 Re8 O-O d5 Nxe5 Nxe5
15024 <LCZero_05ID107&#40;0&#41;&#58; info depth 16 nodes 380 nps 344 score cp -4 winrate 48.63% time 1102 pv Nbd2 h6 h3 Re8 O-O d5 Nxe5 Nxe5
15692 <LCZero_05ID107&#40;0&#41;&#58; info depth 17 nodes 679 nps 383 score cp -2 winrate 49.26% time 1770 pv Nbd2 h6 h3 Re8 O-O d5 Nxe5 Nxe5 Bxe5
16854 <LCZero_05ID107&#40;0&#41;&#58; info depth 18 nodes 1217 nps 415 score cp -1 winrate 49.61% time 2932 pv Nbd2 h6 h3 Re8 O-O d5 Nxe5 Nxe5 Bxe5 c6
18854 <LCZero_05ID107&#40;0&#41;&#58; info depth 19 nodes 2186 nps 443 score cp 0 winrate 49.73% time 4931 pv Nbd2 h6 h3 Re8 O-O d5 Nxe5 Nxe5 Bxe5 c6
22113 <LCZero_05ID107&#40;0&#41;&#58; info depth 20 nodes 3930 nps 480 score cp 0 winrate 49.79% time 8192 pv Nbd2 h6 h3 Re8 O-O d5 Nxe5 Nxe5 Bxe5 c6
27335 <LCZero_05ID107&#40;0&#41;&#58; info depth 21 nodes 7058 nps 526 score cp 0 winrate 49.94% time 13413 pv Nbd2 h6 h3 Re8 O-O Bf8 a3 d5 b4 a6
35992 <LCZero_05ID107&#40;0&#41;&#58; info depth 22 nodes 12683 nps 575 score cp 0 winrate 50.06% time 22070 pv Nbd2 h6 h3 Re8 O-O Bf8 a3 d5 b4 a6 c4 d4
41927 <LCZero_05ID107&#40;0&#41;&#58; info string    g4 ->       1   &#40;V&#58; 38.96%) &#40;N&#58;  0.17%) PV&#58; g4 Bxg4
41928 <LCZero_05ID107&#40;0&#41;&#58; info string   Nd4 ->       1   &#40;V&#58; 40.22%) &#40;N&#58;  0.09%) PV&#58; Nd4 exd4
41928 <LCZero_05ID107&#40;0&#41;&#58; info string   Kd2 ->       1   &#40;V&#58; 40.48%) &#40;N&#58;  0.13%) PV&#58; Kd2 h6
41929 <LCZero_05ID107&#40;0&#41;&#58; info string  Bxe5 ->       1   &#40;V&#58; 41.88%) &#40;N&#58;  0.05%) PV&#58; Bxe5 dxe5
41929 <LCZero_05ID107&#40;0&#41;&#58; info string   Bd4 ->       2   &#40;V&#58; 31.50%) &#40;N&#58;  0.08%) PV&#58; Bd4 exd4 exd4
41930 <LCZero_05ID107&#40;0&#41;&#58; info string  Nxe5 ->       2   &#40;V&#58; 38.98%) &#40;N&#58;  0.09%) PV&#58; Nxe5 dxe5 O-O
41930 <LCZero_05ID107&#40;0&#41;&#58; info string   Rf1 ->       2   &#40;V&#58; 41.86%) &#40;N&#58;  0.21%) PV&#58; Rf1 h6 Nbd2
41930 <LCZero_05ID107&#40;0&#41;&#58; info string   Ng5 ->       2   &#40;V&#58; 42.89%) &#40;N&#58;  0.17%) PV&#58; Ng5 d5 Nf3
41931 <LCZero_05ID107&#40;0&#41;&#58; info string   Rg1 ->       2   &#40;V&#58; 42.96%) &#40;N&#58;  0.22%) PV&#58; Rg1 h6 h3
41931 <LCZero_05ID107&#40;0&#41;&#58; info string   Nh4 ->       2   &#40;V&#58; 44.55%) &#40;N&#58;  0.16%) PV&#58; Nh4 d5 Nf3
41932 <LCZero_05ID107&#40;0&#41;&#58; info string   Kf1 ->       4   &#40;V&#58; 44.01%) &#40;N&#58;  0.33%) PV&#58; Kf1 h6 h3 Re8 Kg1
41932 <LCZero_05ID107&#40;0&#41;&#58; info string    g3 ->       8   &#40;V&#58; 43.85%) &#40;N&#58;  0.61%) PV&#58; g3 Bh3 Nbd2 h6 Bf1 Bxf1
41933 <LCZero_05ID107&#40;0&#41;&#58; info string   Ng1 ->       8   &#40;V&#58; 46.40%) &#40;N&#58;  0.37%) PV&#58; Ng1 d5 Nf3 e4 dxe4 dxe4 Nd4 Nxd4
41933 <LCZero_05ID107&#40;0&#41;&#58; info string    b4 ->       9   &#40;V&#58; 44.55%) &#40;N&#58;  0.57%) PV&#58; b4 Nxb4 O-O c5 a3 Nc6 Nbd2 h6
41934 <LCZero_05ID107&#40;0&#41;&#58; info string    h4 ->      11   &#40;V&#58; 44.45%) &#40;N&#58;  0.72%) PV&#58; h4 h6 Nbd2 Re8 a3 a5 a4 d5
41934 <LCZero_05ID107&#40;0&#41;&#58; info string   Ba3 ->      11   &#40;V&#58; 45.71%) &#40;N&#58;  0.54%) PV&#58; Ba3 h6 O-O Re8 h3 d5 Bxe7
41935 <LCZero_05ID107&#40;0&#41;&#58; info string   Qc1 ->      12   &#40;V&#58; 45.93%) &#40;N&#58;  0.58%) PV&#58; Qc1 h6 O-O Re8 Nbd2 Bf8 h3
41935 <LCZero_05ID107&#40;0&#41;&#58; info string   Bc1 ->      13   &#40;V&#58; 46.25%) &#40;N&#58;  0.59%) PV&#58; Bc1 d5 Bb2 e4 dxe4 dxe4 Nd4 Nxd4
41936 <LCZero_05ID107&#40;0&#41;&#58; info string   Bf1 ->      19   &#40;V&#58; 47.69%) &#40;N&#58;  0.55%) PV&#58; Bf1 h6 Be2 Re8 h3 d5 Nxe5 Nxe5
41937 <LCZero_05ID107&#40;0&#41;&#58; info string   Bc3 ->      20   &#40;V&#58; 48.25%) &#40;N&#58;  0.51%) PV&#58; Bc3 h6 h3 d5 Nxe5 Nxe5 Bxe5
41937 <LCZero_05ID107&#40;0&#41;&#58; info string   Na3 ->      29   &#40;V&#58; 47.33%) &#40;N&#58;  0.82%) PV&#58; Na3 h6 O-O Re8 h3 a6 Nc4
41938 <LCZero_05ID107&#40;0&#41;&#58; info string    c3 ->      30   &#40;V&#58; 46.87%) &#40;N&#58;  1.18%) PV&#58; c3 h6 Nbd2 d5 O-O Re8 b4 a5
41938 <LCZero_05ID107&#40;0&#41;&#58; info string   Qd2 ->      47   &#40;V&#58; 47.18%) &#40;N&#58;  1.68%) PV&#58; Qd2 h6 Nc3 d5 a3 d4 exd4 exd4 Ne4
41939 <LCZero_05ID107&#40;0&#41;&#58; info string  Nfd2 ->      53   &#40;V&#58; 47.40%) &#40;N&#58;  1.77%) PV&#58; Nfd2 d5 O-O Re8 h3 h6 a3 Bd6 c4
41939 <LCZero_05ID107&#40;0&#41;&#58; info string    e4 ->     150   &#40;V&#58; 48.37%) &#40;N&#58;  3.50%) PV&#58; e4 d5 Nbd2 d4 a3 h6 h3 Nh5
41940 <LCZero_05ID107&#40;0&#41;&#58; info string    a4 ->     322   &#40;V&#58; 49.37%) &#40;N&#58;  4.38%) PV&#58; a4 h6 Nbd2 Re8 h3 d5 Nxe5 Nxe5
41940 <LCZero_05ID107&#40;0&#41;&#58; info string    d4 ->     394   &#40;V&#58; 48.77%) &#40;N&#58;  7.72%) PV&#58; d4 e4 Nfd2 d5 c4 Nb4 a3 Nd3+ Bxd3 exd3 O-O c6 h3 Re8 Nc3 Bf5 Qf3
41941 <LCZero_05ID107&#40;0&#41;&#58; info string   Nc3 ->     452   &#40;V&#58; 48.90%) &#40;N&#58;  8.28%) PV&#58; Nc3 d5 O-O h6 d4 e4 Nd2 Re8 h3
41941 <LCZero_05ID107&#40;0&#41;&#58; info string    a3 ->     960   &#40;V&#58; 50.01%) &#40;N&#58;  6.96%) PV&#58; a3 h6 Nbd2 Re8 h3 d5 Nxe5 Nxe5 Bxe5 Bd6
41942 <LCZero_05ID107&#40;0&#41;&#58; info string    c4 ->    1322   &#40;V&#58; 50.25%) &#40;N&#58;  6.52%) PV&#58; c4 h6 Nc3 d5 cxd5 Nxd5 Nxd5 Qxd5 O-O Be6 d4 e4 Nd2 Rad8
41943 <LCZero_05ID107&#40;0&#41;&#58; info string    h3 ->    3075   &#40;V&#58; 50.13%) &#40;N&#58; 17.30%) PV&#58; h3 h6 Nbd2 Re8 O-O Bf8 a3 d5 b4 a6 c4
41943 <LCZero_05ID107&#40;0&#41;&#58; info string   O-O ->    3296   &#40;V&#58; 50.16%) &#40;N&#58; 17.47%) PV&#58; O-O h6 Nbd2 Re8 h3 Bf8 a3 d5 b4 a6 c4 d4
41944 <LCZero_05ID107&#40;0&#41;&#58; info string  Nbd2 ->    6525   &#40;V&#58; 50.50%) &#40;N&#58; 15.67%) PV&#58; Nbd2 h6 h3 Re8 O-O Bf8 a3 d5 b4 a6 c4 d4
41944 <LCZero_05ID107&#40;0&#41;&#58; info depth 22 nodes 16787 nps 599 score cp 0 winrate 50.12% time 28016 pv Nbd2 h6 h3 Re8 O-O Bf8 a3 d5 b4 a6 c4 d4
41945 <LCZero_05ID107&#40;0&#41;&#58; bestmove b1d2

[Laskos]

On tactical test suite, also no significant progress. Let's hope for now that move selection in the openings will change this real stalling (that in self-games was artificial). It surely was stuck with some local optima in the openings, and the whole training process was flawed. If not this is the issue, a larger block/filters network will be required.

The change with opening move selection is only going to affect matches, which should yield more accurate self-play Elo ratings in the future. It did not and will not affect the training process at all, since training games always had the full variety in move selection. The most likely candidate for the stall is really the network size, which will be increased within a few days if all goes as planned.

Ah, it affects only matches, not the training. But if I used in matches a varied opening suite, it doesn't matter, right? This patch might even weaken a bit the engine in matches, right? Yes, self-matches will show more real Elo. Thus, the stalling seems to show that a larger network size is needed, but let's see yet.

[Werewolf]

I'm getting stunning results with 107 ID and the v0.5 network.

A week ago I was running matches with the Master dedicated chess computer which is about 2150 elo. It nearly always won.

Now LCZ 107 beat the Master twice even though I let the Master think much longer, and it beat the Mephisto London 68030 (66 MHz Tournament machine) twice as well!

The latter must be close to 2400 elo.

[AdminX]

I'm getting stunning results with 107 ID and the v0.5 network.

A week ago I was running matches with the Master dedicated chess computer which is about 2150 elo. It nearly always won.

Now LCZ 107 beat the Master twice even though I let the Master think much longer, and it beat the Mephisto London 68030 (66 MHz Tournament machine) twice as well!

The latter must be close to 2400 elo.

What TC are you testing with?

[mhull]

Indeed. Leela's output is an expected score, not actually a win%.

The wording on the site has since been changed, it seems, although now it includes this "50%=draw" bit in the legend, which caused some debate in the discord.

Probably should just say "50%=equal chances" or some such thing ("expected score" is pretty self-explanatory, so could probably do away with the legend altogether), but not a big deal.

Last net to pass is actually fairly reasonable now. It'll be interesting to see where we are a week from now (it's not even been a week since the last big bug was fixed).

Something is clearly wrong with the probability

After 1.f3 e5 2.g4
I get the right move but the following message
Leela thinks her expected score is 97.17%.

How is it possible that is not 100%?

Because the expectation is based on self-play, the typical opponent (self) is not a grandmaster.

[Werewolf]

I'm getting stunning results with 107 ID and the v0.5 network.

A week ago I was running matches with the Master dedicated chess computer which is about 2150 elo. It nearly always won.

Now LCZ 107 beat the Master twice even though I let the Master think much longer, and it beat the Mephisto London 68030 (66 MHz Tournament machine) twice as well!

The latter must be close to 2400 elo.

What TC are you testing with?

I give LCZ ID 107 30 seconds per move. It usually takes 15-20 seconds to move on this setting.

The dedicated machines had an hour for the whole game. They took about 5x longer per move.

[Uri Blass]

On tactical test suite, also no significant progress. Let's hope for now that move selection in the openings will change this real stalling (that in self-games was artificial). It surely was stuck with some local optima in the openings, and the whole training process was flawed. If not this is the issue, a larger block/filters network will be required.

The change with opening move selection is only going to affect matches, which should yield more accurate self-play Elo ratings in the future. It did not and will not affect the training process at all, since training games always had the full variety in move selection. The most likely candidate for the stall is really the network size, which will be increased within a few days if all goes as planned.

I think that the most candidate for the stall is the fact that LCzero insist to do the same stupid things that alpha zero did instead of trying to improve.

LCzero is weak in seeing mates and they do not consider it as a problem for some reason.

It can miss mate in 1 at least in easy mode.
What about adding a check if there is mate in 1 to LCzero in every position before making playout against itself?

What about simply adding alphabeta to LCzero?

I believe that simple alphabeta when the evaluation at the leaves is simply
expected score at easy mode can do LCzero stronger.

[Jhoravi]

On tactical test suite, also no significant progress. Let's hope for now that move selection in the openings will change this real stalling (that in self-games was artificial). It surely was stuck with some local optima in the openings, and the whole training process was flawed. If not this is the issue, a larger block/filters network will be required.

The change with opening move selection is only going to affect matches, which should yield more accurate self-play Elo ratings in the future. It did not and will not affect the training process at all, since training games always had the full variety in move selection. The most likely candidate for the stall is really the network size, which will be increased within a few days if all goes as planned.

I think that the most candidate for the stall is the fact that LCzero insist to do the same stupid things that alpha zero did instead of trying to improve.

LCzero is weak in seeing mates and they do not consider it as a problem for some reason.

It can miss mate in 1 at least in easy mode.
What about adding a check if there is mate in 1 to LCzero in every position before making playout against itself?

What about simply adding alphabeta to LCzero?

I believe that simple alphabeta when the evaluation at the leaves is simply
expected score at easy mode can do LCzero stronger.

What about LCZero that trains only Static positions that has no attacks and checks and just let the quiescence search take care of the tactics?