LCZero: Progress and Scaling. Relation to CCRL Elo

[Albert Silver]

At the moment, I play two gauntlets on my third notebook: Komodo 5 vs. Leela Cuda with default settings and Komodo 5 vs. Leela Cuda with the older Clop-Settings (cpuct=3.168, fpu reduction=-0.0683). With 5'+3“ thinking time. I want to play at least 200 games, each. When this is done, I will try your new Clop-tuned values, too. It is interesting, that both Clop-tunings gave nearly the same cpuct-value (around 3.1). Only the fpu-reduction is very different.

Stefan (SPCC)

Yes, it surprised me too, and I gave it a lot of games to show any changes but it never did, and stuck to 0.14 to 0.19 after 400 games or so.

[Laskos]

Thanks Albert for your great, lengthy work. I will put your settings compared to default against Arasan 20.5 in a gauntlet of 1600 ultra-fast games total, to see if the settings help. One of the problems of this LCzero in general is that it is scaling in some uncharted ways, and the settings may vary with time control, but if I will get something with your settings in ultra-fast games, then some confidence of fast testing may open new ways to optimize it. I will post the results in some maybe 7 hours, it still takes some time.

I played 1500 games, probably I typed 1500 instead of 1600, but it doesn't matter, the result seems conclusive:

Code: Select all

    Program                            Score      %      Elo    +   -    Draws

  1 Arasan 20.5                    : 997.0/1500  66.5    - 60   16  16   21.2 %
  2 Lc0 357 default                : 272.5/ 750  36.3     -38   23  23   20.7 %
  3 Lc0 357 Albert CLOP            : 230.5/ 750  30.7     -81   23  23   21.7 %

So, the new CLOP settings are 43 +/- 32 Elo points weaker than the default settings at 0.1 s/move. Probably settings depend on time control (and hardware, but we have identical GTX 1060 6GB), as I myself observed. It's a pain playing with them.

You're aware that the engine already has a 100ms (0.1s) overhead in the UCI settings, so basically you are running instant moves. I don't doubt your results, but I think you must be aware already that Leela does not scale in any way like normal A/B engines. My testing was already at an average 1-2 seconds per move. For me an engine is for analyzing and playing, and I don't know anyone who does either at a pace of microseconds thus I so see no point tuning for them.

Yes, sure, I just cannot collect enough games in some reasonable amount of time with longer TC games. I don't think that overhead here matters, it depends on UI/GUI, and Lc0 at 0.1s/move is competitive with Arasan 20.5, which doesn't have overhead, as it is competitive at much longer TC (sure, Lc0 improves, but this is expected). The moves are surely not instant moves, and it seems as handled by UI, they are normal 100ms moves. The problem I observed with much lower number of games but pretty long TC is that the results vary significantly with TC, so your results at 1'+ 1'' might be different at 10'+ 10''. But my samples were small and I got annoyed with these small samples and jumping up and down results, when optimizing on tactical test-suites.

[Laskos]

It seems the latest cuDNN binary from here:

https://crem.xyz/lc0/

is stronger by some 50+ Elo points than the previous stable one. NPS are also significantly higher. Tested in games at 1s/move, against Arasan 20.5. Now compared to the official v0.10 there is huge gap, maybe up to 200 Elo points.

[JJJ]

It seems the latest cuDNN binary from here:

https://crem.xyz/lc0/

is stronger by some 50+ Elo points than the previous stable one. NPS are also significantly higher. Tested in games at 1s/move, against Arasan 20.5. Now compared to the official v0.10 there is huge gap, maybe up to 200 Elo points.

I have 10k nodes with it with my geforce 1060 , it is 10 times faster than the normal version without cudnn.

Is leela really better at 5+ 10 sec than 1 min + 1 sec ? It seems not so obvious to me.

[Milos]

Is leela really better at 5+ 10 sec than 1 min + 1 sec ? It seems not so obvious to me.

How it wouldn't, 5+10s is 10s per move essentially, and 1min+1s is 5-6s per move for first 20 moves and then it drops.

[JJJ]

Is leela really better at 5+ 10 sec than 1 min + 1 sec ? It seems not so obvious to me.

How it wouldn't, 5+10s is 10s per move essentially, and 1min+1s is 5-6s per move for first 20 moves and then it drops.

I meant, does he scales better than others engines ? Seems to me the winrate doesn't improve at this time control, but I can't be sure yet.

[Damir]

Why is no new Leela version released here : https://github.com/glinscott/leela-chess/releases

New versions keep beeing released here: https://crem.xyz/lc0/ but I can not use any of them cause are not compatible with my hardware.

Is CPU/GPU version no longer worked on ?

[CMCanavessi]

Why is no new Leela version released here : https://github.com/glinscott/leela-chess/releases

New versions keep beeing released here: https://crem.xyz/lc0/ but I can not use any of them cause are not compatible with my hardware.

Is CPU/GPU version no longer worked on ?

It's not, lc0 will replace lczero in the near future (a week or so), and now it supports opencl too, not only cuda; but you need to compile it yourself. When it's made official, it will of course be released as usual (probably as v1.x or something)

[pohl4711]

It seems the latest cuDNN binary from here:

https://crem.xyz/lc0/

is stronger by some 50+ Elo points than the previous stable one. NPS are also significantly higher. Tested in games at 1s/move, against Arasan 20.5. Now compared to the official v0.10 there is huge gap, maybe up to 200 Elo points.

Wow! You are right. On my mobile GPU (GTX 950M), the latest version (180530) is exactly +25% faster, than the older versions! Thanx for this information. Very cool speedup. No, I have 2600 rollouts/s in the starting position until depth 37. Nice!!!

Stefan (SPCC)

[yanquis1972]

been watching a 1'1 match between leela & SF8 on one core, & it's transparently clear that, while tactics are an obvious issue, leela's woeful misevaluation of endgames is bleeding a lot of wins (& draws). i don't really understand how, after 15 million games, it could be so bad, but she has repeatedly dissolved into drawn endgames she thinks are +2 or +3. KRB vs KR is the last example off the top of my ahead, & i think the game eval was something like +2.50.

is there any concrete evidence this is improving?