LCZero: Progress and Scaling. Relation to CCRL Elo

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
CMCanavessi
Posts: 835
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by CMCanavessi » Sun May 27, 2018 10:46 pm

Graham Banks wrote:
Sat May 26, 2018 10:08 pm
http://www.computerchess.org.uk/ccrl/40 ... 4-bit_w323

Leela Chess 0.10 64-bit w323 #110‑111 (2651 +19 −19)
2100 elo engine!!!! :roll:
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

Albert Silver
Posts: 2861
Joined: Wed Mar 08, 2006 8:57 pm
Location: Rio de Janeiro, Brazil

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Albert Silver » Mon May 28, 2018 3:55 am

CMCanavessi wrote:
Sun May 27, 2018 10:46 pm
Graham Banks wrote:
Sat May 26, 2018 10:08 pm
http://www.computerchess.org.uk/ccrl/40 ... 4-bit_w323

Leela Chess 0.10 64-bit w323 #110‑111 (2651 +19 −19)
2100 elo engine!!!! :roll:
It is. Around 2100. You are not really going to nitpick over a piddly 550 Elo are you? Mind you, I think that was just using the CPU version too.
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."

Werewolf
Posts: 1197
Joined: Thu Sep 18, 2008 8:24 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Werewolf » Mon May 28, 2018 7:38 am

Has Leela hit a plateau? There doesn't seem to be progress anymore.

Also any news on the bigger network?

Milos
Posts: 3387
Joined: Wed Nov 25, 2009 12:47 am

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Milos » Mon May 28, 2018 11:46 am

Werewolf wrote:
Mon May 28, 2018 7:38 am
Has Leela hit a plateau? There doesn't seem to be progress anymore.

Also any news on the bigger network?
It hit plateau already early with 15x192 network around 237. So far this was mostly regression and very small improvement (since network 320 basically).
20x256 won't bring anything since net gain is smaller than computation slowdown.

yanquis1972
Posts: 1762
Joined: Tue Jun 02, 2009 10:14 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by yanquis1972 » Mon May 28, 2018 1:18 pm

Werewolf wrote:
Mon May 28, 2018 7:38 am
Has Leela hit a plateau? There doesn't seem to be progress anymore.

Also any news on the bigger network?
don't know any details, but superficially it looks like self-play elo has plateaued but benchmarks against SF show progress continuing. the latest net looks like the strongest by that criteria & seems like it'd be a good one for thorough testing at longer time controls. think the CCRL rating on a GTX 1060 would surprise a lot of people one way or another if it was tested properly.

https://docs.google.com/spreadsheets/d/ ... li=1#gid=0
https://docs.google.com/spreadsheets/d/ ... edit#gid=0

User avatar
Leto
Posts: 2028
Joined: Thu May 04, 2006 1:40 am
Location: Dune

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Leto » Mon May 28, 2018 2:41 pm

According to this chart it seems Leela is now almost as strong as Rodent 3, so it seems to be making progress:
https://docs.google.com/spreadsheets/d/ ... edit#gid=0

Werewolf
Posts: 1197
Joined: Thu Sep 18, 2008 8:24 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Werewolf » Mon May 28, 2018 8:25 pm

To me it looks like stalling. I wonder if this can be resurrected.

User avatar
CMCanavessi
Posts: 835
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by CMCanavessi » Mon May 28, 2018 8:33 pm

Werewolf wrote:
Mon May 28, 2018 8:25 pm
To me it looks like stalling. I wonder if this can be resurrected.
How can you "resurrect" something that's not dead? :roll:
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

yanquis1972
Posts: 1762
Joined: Tue Jun 02, 2009 10:14 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by yanquis1972 » Mon May 28, 2018 8:45 pm

yeah idk, maybe look at it harder. it's not gonna be 50 elo/day or whatever, that part of the curve has sailed. it's clearly not stalled. 321-351 is tiny a blip & there's measurable growth.

User avatar
Laskos
Posts: 9467
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Laskos » Tue May 29, 2018 11:56 pm

Werewolf wrote:
Mon May 28, 2018 8:25 pm
To me it looks like stalling. I wonder if this can be resurrected.
From NN342 to NN352 it indeed seems to be stalling against an A/B engine Arasan 20.5. Here is the results of gauntlet:

Code: Select all

Games Completed = 1600 of 1600 (Avg game length = 13.001 sec)
Settings = Gauntlet/64MB/100ms per move/M 9000cp for 30 moves, D 150 moves/EPD:C:\LittleBlitzer\3moves_GM_04.epd(817)
Time = 29352 sec elapsed, 0 sec remaining
 1.  Arasan 20.5              	1062.5/1600	872-347-381  	(L: m=347 t=0 i=0 a=0)	(D: r=297 i=55 f=7 s=9 a=13)	(tpm=107.6 d=16.21 nps=1682392)
 2.  Lc0 NN342                	269.0/800	172-434-194  	(L: m=429 t=0 i=5 a=0)	(D: r=149 i=26 f=4 s=6 a=9)	(tpm=108.8 d=1.25 nps=1871)
 3.  Lc0 NN352                	268.5/800	175-438-187  	(L: m=436 t=0 i=2 a=0)	(D: r=148 i=29 f=3 s=3 a=4)	(tpm=108.6 d=1.22 nps=1700)
Lc0 is the cuDNN version with default settings, I am tired of fiddling with its parameters, with performances varying at time controls, and it seems anyway there are some serious bugs.

To see the scaling, I played matches of 800 games each at 0.1s/move, 0.4s/move, 1.6s/move, in total 4 doublings, of Lc0 NN352 on GTX 1060 6GB against Arasan 20.5 on one core (3061 CCRL 40/4' Elo points on one core). In fact the time used per move was taken from what LittleBlitzer reports, and it is not exactly what I set there. The CCRL performance of Lc0 cuDNN ID352 at three time controls are:

0.109 s/move --> 2943 Elo points
0.344 s/move --> 3015 Elo points
1.215 s/move --> 3091 Elo points

The fitted function as CCRL Elo performance as function of time control (scaling) of Lc0 with time per move is

Code: Select all

CCRL Elo of cuDNN Lc0 = 3079.51 + 61.3627 * ln(seconds per move) 
on GTX 1060. 
I also assumed that Arasan 20.5 scales as a standard A/B engine. The fitted intuitive function of scaling with two parameters correlates 0.99999 with the three data points, and it is not an overfit. Here is the plot:

Image

So, at LTC, cuDNN Lc0 ID352 is above 3300 CCRL Elo points by this extrapolation, which confirms my earlier results, when playing games against Stockfish (but with very weak accumulated statistic). Here the weak point is the extrapolation itself, but I prefer that. For a top GTX 1080 Ti GPU, add some 70 Elo points to these results, maybe more.

Post Reply