Page 1 of 3

Scaling of Lc0 at high Leela Ratio

Posted: Tue Nov 27, 2018 7:54 am
by Laskos
"Leela Ratio" notion is overused and often misleading, but in general terms, with my very strong RTX 2070 GPU and mediocre CPU, an OCed 4-core i7-4790, I have a high "Leela Ratio". "Leela Ratio" is an abused term, as it depends on time control (Lc0 gains speed steadily with TC), and it only relates to NPS. But we don't know what the effective speed-up is on 64 threads used to get 70 million NPS reported for SF8 in A0 paper, we don't know what the effective speed-up is on 4 TPUs used to get 80 thousand NPS for A0. Also, we don't know on what position these speeds were achieved.

But let's say my "effective Leela Ratio" is about ~ 2-3 in the games I performed, depending somewhat on time control and positions. This is pretty high. In some range of time controls, Lc0 ID11261 is stronger than even recently released Stockfish 10 dev on all 4 cores, being the strongest engine overall on my PC. But only on some range of usual time controls. I tried to check the scaling of Lc0 ID11261 compared to SF10. And it came as non-trivial.

Games:
Lc0 ID11261 on 2 CPU threads and RTX 2070 GPU.
SF10 on 4 threads on 4 i7-4790 OCed cores.
No Syzygy are used during the games, but the games are adjudicated in Cutechess-Cli by 6-men Syzygy. I used no Syzygy during the games because during long tests, my machine with 16GB RAM can have instabilities using 6-men Syzygy from SSD. I use no swap file, because when using 6-men Syzygy during the games, most other applications are moved from RAM to HDD swap file, and the PC is barely moving, aside the playing chess engines. But without a swap file, in 1-2 days of tests with Syzygy, my PC can start giving warnings and even crash.

I have the following results at different time controls Lc0 vs SF10 (not very many games, though):

0.25m + 0.25s
Score of lc0_v19_11261 vs SF_10: 19 - 26 - 55 [0.465] 100
Elo difference: -24.36 +/- 45.86

1m + 1s
Score of lc0_v19_11261 vs SF_10: 20 - 17 - 63 [0.515] 100
Elo difference: 10.43 +/- 41.55
Finished match

4m + 4s
Score of lc0_v19_11261 vs SF_10: 7 - 12 - 81 [0.475] 100
Elo difference: -17.39 +/- 29.59
Finished match

16m + 16s
Score of lc0_v19_11261 vs SF_10: 0 - 4 - 16 [0.400] 20
Elo difference: -70.44 +/- 64.16
Finished match


Time control is going in factors of 4x from 0.25 minutes + 0.25 seconds increment, up to 16 minutes + 16 seconds increment time controls. In this, latter LTC test, the nodes per move are about 0.5-1 million for Lc0, even higher in the first phase of the game, where more time per move is often allotted.

The plot of these performances is here:

Image


We see that Lc0 scales better than SF 10 only at short TC, and is the strongest engine (stronger than SF 10 on 4 cores) in the time control range of about 30s + 0.5s to 2m + 2s. But the scaling to longer TC is worse than that of SF 10, becoming significantly weaker than SF 10 at 16m + 16s. So, to LTC, Lc0 seems to scale worse than SF 10. Interesting that for Komodo MCTS I observed the same kind of behavior. Either both MCTS searches are tuned to fairly short time controls in both cases, or MCTS generally scales worse to LTC than finely tuned and pruned AB.

Re: Scaling of Lc0 at high Leela Ratio

Posted: Tue Nov 27, 2018 8:23 am
by Laskos
To observe that I specified "at high Leela Ratio". At usually used and smaller Leela Ratios, the graph will shift significantly down and to the right, the effect of worse scaling of Lc0 to LTC being harder to observe (needs really long TC).

Re: Scaling of Lc0 at high Leela Ratio

Posted: Tue Nov 27, 2018 12:28 pm
by Javier Ros
Very interesting experiment, thanks for sharing.

I have read in lc0 forum that somebody when using powerful GPU has put nncache=50000 instead of default nncache=20000

see for example

https://groups.google.com/forum/#!topic ... 0sJNuFMLPE

Maybe you can try it at longer time controls.

Re: Scaling of Lc0 at high Leela Ratio

Posted: Tue Nov 27, 2018 1:52 pm
by shrapnel
Looks like Lc0 is nowhere near AlphaZero levels.
Wish some AlphaZero team member would give a few anonymous tips the lc0 people, who seem to be floundering.
The wait goes on...
Oh well, let the Stockfishes and Komodos have their days in the Sun....while it lasts... mainly due to an abominable lack of interest in chess shown by the Google Teams.
But, I sense, that the Day is not far when the alpha-beta engines will go the way of the Dinosaurs...
Which will deliver the coup de grâce, lco, AlphaZero or a third, as yet unreleased Engine, remains to be seen...

Re: Scaling of Lc0 at high Leela Ratio

Posted: Tue Nov 27, 2018 3:37 pm
by Laskos
Javier Ros wrote:
Tue Nov 27, 2018 12:28 pm
Very interesting experiment, thanks for sharing.

I have read in lc0 forum that somebody when using powerful GPU has put nncache=50000 instead of default nncache=20000

see for example

https://groups.google.com/forum/#!topic ... 0sJNuFMLPE

Maybe you can try it at longer time controls.
Yes, I also have NNCacheSize from default 200000 to 2000000.

Re: Scaling of Lc0 at high Leela Ratio

Posted: Tue Nov 27, 2018 4:19 pm
by yanquis1972
curious about test30 scaling; imo it should be much closer to A0, at least at present.

using DM openings, 128MB hash, 3 cores/~7MN/s for SF8 & default settings for Lc0 w/ a GTX 1080 it scored 2W-4L-18D (-28 elo) @ 0+12s in the first batch of 24 games i ran. should be very roughly equivalent to 1s/move A0 matches.

Re: Scaling of Lc0 at high Leela Ratio

Posted: Tue Nov 27, 2018 6:49 pm
by megamau
Including the error bars in the graph, the shape of the scaling could easily be a parabola in the other direction (i.e. with the minimum where you have the maximum). Also, comparison between Alphazero and Leela should consider that the match was with SF8, not with SF10 (more than 100 elo difference).


TLDR: small sample size.

Re: Scaling of Lc0 at high Leela Ratio

Posted: Tue Nov 27, 2018 7:17 pm
by Laskos
megamau wrote:
Tue Nov 27, 2018 6:49 pm
Including the error bars in the graph, the shape of the scaling could easily be a parabola in the other direction (i.e. with the minimum where you have the maximum). Also, comparison between Alphazero and Leela should consider that the match was with SF8, not with SF10 (more than 100 elo difference).


TLDR: small sample size.
Yes, but I have some a bit earlier tests showing similar results, but with even smaller samples. You have to know some statistics to appreciate the shape and the likelihood that Lc0 scales worse in my conditions than SF10 to LTC, which is as an estimate, above 90%. And combined with my previous, even more scanty results, above 95%. Otherwise I wouldn't have posted until I don't get something fairly statistically conclusive.

Re: Scaling of Lc0 at high Leela Ratio

Posted: Wed Nov 28, 2018 4:25 am
by Uri Blass
shrapnel wrote:
Tue Nov 27, 2018 1:52 pm
Looks like Lc0 is nowhere near AlphaZero levels.
Wish some AlphaZero team member would give a few anonymous tips the lc0 people, who seem to be floundering.
The wait goes on...
Oh well, let the Stockfishes and Komodos have their days in the Sun....while it lasts... mainly due to an abominable lack of interest in chess shown by the Google Teams.
But, I sense, that the Day is not far when the alpha-beta engines will go the way of the Dinosaurs...
Which will deliver the coup de grâce, lco, AlphaZero or a third, as yet unreleased Engine, remains to be seen...
I totally dislike the MCTS algorithm because it means that you cannot see simple things like the fact that a move is leading to mate in 2 for the opponent.

Komodo MCTS did not see mate in 2 in the only game that it lost in TCEC when it played 44...Kxf7

Every human who is not a beginner is going to see that there is a forced mate after 44...Kxf7 but Komodo played it with 29.91 score after 26 seconds with 185734 nodes.
I guess that A0 also show a similiar behaviour and it is clearly not the way that humans think.

http://legacy-tcec.chessdom.com/archive ... i=4&ga=183

I do not believe that the reason for AlphaZero's good results is the fact that they did not use the alphabeta and I believe that alphaZero can be stronger if they fix it to use a better algorithm that can see forced mates with a mate score in the same way that humans see.

Maybe it is possible to have some combination of alphabeta and MCTS but I find the pure MCTS to be an extremely stupid algorithm if it cannot see mate scores even for mates that humans can easily calculate.

Re: Scaling of Lc0 at high Leela Ratio

Posted: Wed Nov 28, 2018 9:20 am
by jp
Uri Blass wrote:
Wed Nov 28, 2018 4:25 am
I totally dislike the MCTS algorithm because it means that you cannot see simple things like the fact that a move is leading to mate in 2 for the opponent.

Komodo MCTS did not see mate in 2 in the only game that it lost in TCEC when it played 44...Kxf7

I do not believe that the reason for AlphaZero's good results is the fact that they did not use the alphabeta and I believe that alphaZero can be stronger if they fix it to use a better algorithm that can see forced mates with a mate score in the same way that humans see.
But Komodo MCTS is using real MCTS for playing.
A0 & Lc0 are not, so you cannot really compare.