Scaling of Lc0 at high Leela Ratio

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Laskos
Posts: 9456
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Scaling of Lc0 at high Leela Ratio

Post by Laskos » Tue Nov 27, 2018 7:54 am

"Leela Ratio" notion is overused and often misleading, but in general terms, with my very strong RTX 2070 GPU and mediocre CPU, an OCed 4-core i7-4790, I have a high "Leela Ratio". "Leela Ratio" is an abused term, as it depends on time control (Lc0 gains speed steadily with TC), and it only relates to NPS. But we don't know what the effective speed-up is on 64 threads used to get 70 million NPS reported for SF8 in A0 paper, we don't know what the effective speed-up is on 4 TPUs used to get 80 thousand NPS for A0. Also, we don't know on what position these speeds were achieved.

But let's say my "effective Leela Ratio" is about ~ 2-3 in the games I performed, depending somewhat on time control and positions. This is pretty high. In some range of time controls, Lc0 ID11261 is stronger than even recently released Stockfish 10 dev on all 4 cores, being the strongest engine overall on my PC. But only on some range of usual time controls. I tried to check the scaling of Lc0 ID11261 compared to SF10. And it came as non-trivial.

Games:
Lc0 ID11261 on 2 CPU threads and RTX 2070 GPU.
SF10 on 4 threads on 4 i7-4790 OCed cores.
No Syzygy are used during the games, but the games are adjudicated in Cutechess-Cli by 6-men Syzygy. I used no Syzygy during the games because during long tests, my machine with 16GB RAM can have instabilities using 6-men Syzygy from SSD. I use no swap file, because when using 6-men Syzygy during the games, most other applications are moved from RAM to HDD swap file, and the PC is barely moving, aside the playing chess engines. But without a swap file, in 1-2 days of tests with Syzygy, my PC can start giving warnings and even crash.

I have the following results at different time controls Lc0 vs SF10 (not very many games, though):

0.25m + 0.25s
Score of lc0_v19_11261 vs SF_10: 19 - 26 - 55 [0.465] 100
Elo difference: -24.36 +/- 45.86

1m + 1s
Score of lc0_v19_11261 vs SF_10: 20 - 17 - 63 [0.515] 100
Elo difference: 10.43 +/- 41.55
Finished match

4m + 4s
Score of lc0_v19_11261 vs SF_10: 7 - 12 - 81 [0.475] 100
Elo difference: -17.39 +/- 29.59
Finished match

16m + 16s
Score of lc0_v19_11261 vs SF_10: 0 - 4 - 16 [0.400] 20
Elo difference: -70.44 +/- 64.16
Finished match


Time control is going in factors of 4x from 0.25 minutes + 0.25 seconds increment, up to 16 minutes + 16 seconds increment time controls. In this, latter LTC test, the nodes per move are about 0.5-1 million for Lc0, even higher in the first phase of the game, where more time per move is often allotted.

The plot of these performances is here:

Image


We see that Lc0 scales better than SF 10 only at short TC, and is the strongest engine (stronger than SF 10 on 4 cores) in the time control range of about 30s + 0.5s to 2m + 2s. But the scaling to longer TC is worse than that of SF 10, becoming significantly weaker than SF 10 at 16m + 16s. So, to LTC, Lc0 seems to scale worse than SF 10. Interesting that for Komodo MCTS I observed the same kind of behavior. Either both MCTS searches are tuned to fairly short time controls in both cases, or MCTS generally scales worse to LTC than finely tuned and pruned AB.

User avatar
Laskos
Posts: 9456
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Scaling of Lc0 at high Leela Ratio

Post by Laskos » Tue Nov 27, 2018 8:23 am

To observe that I specified "at high Leela Ratio". At usually used and smaller Leela Ratios, the graph will shift significantly down and to the right, the effect of worse scaling of Lc0 to LTC being harder to observe (needs really long TC).

Javier Ros
Posts: 181
Joined: Fri Oct 12, 2012 10:48 am
Location: Seville (SPAIN)
Full name: Javier Ros

Re: Scaling of Lc0 at high Leela Ratio

Post by Javier Ros » Tue Nov 27, 2018 12:28 pm

Very interesting experiment, thanks for sharing.

I have read in lc0 forum that somebody when using powerful GPU has put nncache=50000 instead of default nncache=20000

see for example

https://groups.google.com/forum/#!topic ... 0sJNuFMLPE

Maybe you can try it at longer time controls.
The love relationship between a chess engine tester and his computer can be summarized in one sentence:
Until heat do us part.

shrapnel
Posts: 1213
Joined: Fri Nov 02, 2012 8:43 am
Location: New Delhi, India

Re: Scaling of Lc0 at high Leela Ratio

Post by shrapnel » Tue Nov 27, 2018 1:52 pm

Looks like Lc0 is nowhere near AlphaZero levels.
Wish some AlphaZero team member would give a few anonymous tips the lc0 people, who seem to be floundering.
The wait goes on...
Oh well, let the Stockfishes and Komodos have their days in the Sun....while it lasts... mainly due to an abominable lack of interest in chess shown by the Google Teams.
But, I sense, that the Day is not far when the alpha-beta engines will go the way of the Dinosaurs...
Which will deliver the coup de grâce, lco, AlphaZero or a third, as yet unreleased Engine, remains to be seen...
i7 5960X @ 4.1 Ghz, 64 GB G.Skill RipJaws RAM, Twin Asus ROG Strix OC 11 GB Geforce 2080 Tis

User avatar
Laskos
Posts: 9456
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Scaling of Lc0 at high Leela Ratio

Post by Laskos » Tue Nov 27, 2018 3:37 pm

Javier Ros wrote:
Tue Nov 27, 2018 12:28 pm
Very interesting experiment, thanks for sharing.

I have read in lc0 forum that somebody when using powerful GPU has put nncache=50000 instead of default nncache=20000

see for example

https://groups.google.com/forum/#!topic ... 0sJNuFMLPE

Maybe you can try it at longer time controls.
Yes, I also have NNCacheSize from default 200000 to 2000000.

yanquis1972
Posts: 1762
Joined: Tue Jun 02, 2009 10:14 pm

Re: Scaling of Lc0 at high Leela Ratio

Post by yanquis1972 » Tue Nov 27, 2018 4:19 pm

curious about test30 scaling; imo it should be much closer to A0, at least at present.

using DM openings, 128MB hash, 3 cores/~7MN/s for SF8 & default settings for Lc0 w/ a GTX 1080 it scored 2W-4L-18D (-28 elo) @ 0+12s in the first batch of 24 games i ran. should be very roughly equivalent to 1s/move A0 matches.

megamau
Posts: 37
Joined: Wed Feb 10, 2016 5:20 am
Location: Singapore

Re: Scaling of Lc0 at high Leela Ratio

Post by megamau » Tue Nov 27, 2018 6:49 pm

Including the error bars in the graph, the shape of the scaling could easily be a parabola in the other direction (i.e. with the minimum where you have the maximum). Also, comparison between Alphazero and Leela should consider that the match was with SF8, not with SF10 (more than 100 elo difference).


TLDR: small sample size.

User avatar
Laskos
Posts: 9456
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Scaling of Lc0 at high Leela Ratio

Post by Laskos » Tue Nov 27, 2018 7:17 pm

megamau wrote:
Tue Nov 27, 2018 6:49 pm
Including the error bars in the graph, the shape of the scaling could easily be a parabola in the other direction (i.e. with the minimum where you have the maximum). Also, comparison between Alphazero and Leela should consider that the match was with SF8, not with SF10 (more than 100 elo difference).


TLDR: small sample size.
Yes, but I have some a bit earlier tests showing similar results, but with even smaller samples. You have to know some statistics to appreciate the shape and the likelihood that Lc0 scales worse in my conditions than SF10 to LTC, which is as an estimate, above 90%. And combined with my previous, even more scanty results, above 95%. Otherwise I wouldn't have posted until I don't get something fairly statistically conclusive.

Uri Blass
Posts: 8590
Joined: Wed Mar 08, 2006 11:37 pm
Location: Tel-Aviv Israel

Re: Scaling of Lc0 at high Leela Ratio

Post by Uri Blass » Wed Nov 28, 2018 4:25 am

shrapnel wrote:
Tue Nov 27, 2018 1:52 pm
Looks like Lc0 is nowhere near AlphaZero levels.
Wish some AlphaZero team member would give a few anonymous tips the lc0 people, who seem to be floundering.
The wait goes on...
Oh well, let the Stockfishes and Komodos have their days in the Sun....while it lasts... mainly due to an abominable lack of interest in chess shown by the Google Teams.
But, I sense, that the Day is not far when the alpha-beta engines will go the way of the Dinosaurs...
Which will deliver the coup de grâce, lco, AlphaZero or a third, as yet unreleased Engine, remains to be seen...
I totally dislike the MCTS algorithm because it means that you cannot see simple things like the fact that a move is leading to mate in 2 for the opponent.

Komodo MCTS did not see mate in 2 in the only game that it lost in TCEC when it played 44...Kxf7

Every human who is not a beginner is going to see that there is a forced mate after 44...Kxf7 but Komodo played it with 29.91 score after 26 seconds with 185734 nodes.
I guess that A0 also show a similiar behaviour and it is clearly not the way that humans think.

http://legacy-tcec.chessdom.com/archive ... i=4&ga=183

I do not believe that the reason for AlphaZero's good results is the fact that they did not use the alphabeta and I believe that alphaZero can be stronger if they fix it to use a better algorithm that can see forced mates with a mate score in the same way that humans see.

Maybe it is possible to have some combination of alphabeta and MCTS but I find the pure MCTS to be an extremely stupid algorithm if it cannot see mate scores even for mates that humans can easily calculate.

jp
Posts: 821
Joined: Mon Apr 23, 2018 5:54 am

Re: Scaling of Lc0 at high Leela Ratio

Post by jp » Wed Nov 28, 2018 9:20 am

Uri Blass wrote:
Wed Nov 28, 2018 4:25 am
I totally dislike the MCTS algorithm because it means that you cannot see simple things like the fact that a move is leading to mate in 2 for the opponent.

Komodo MCTS did not see mate in 2 in the only game that it lost in TCEC when it played 44...Kxf7

I do not believe that the reason for AlphaZero's good results is the fact that they did not use the alphabeta and I believe that alphaZero can be stronger if they fix it to use a better algorithm that can see forced mates with a mate score in the same way that humans see.
But Komodo MCTS is using real MCTS for playing.
A0 & Lc0 are not, so you cannot really compare.

Post Reply