LCZero: Progress and Scaling. Relation to CCRL Elo

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Laskos »

First, the scaling with time control.
NN with backing up the average score MCTS used in LCZero seems to scale a bit better than Alpha-Beta conventional modern engine like Zurichess. I took Zurichess Appenzeller 64-bit (Elo 1821 CCRL 40/4'), a modern conventional engine of the Zurichess line of engines, which in 2 years became pretty strong, and played 4 matches against yesterday's network 44 for LCZero, at 0.1s/move, 1s/move, 10s/move and 40s/move (a total factor of 400 in time control), and the results show a bit better scaling of LCZero:


LCZero versus Zurichess Appenzeller (Elo 1821 CCRL):
  • 0.1s / move:
    7.5 - 92.5

    1s / move:
    9.0 - 91.0

    10s / move
    11.5 - 88.5

    40s / move
    16.5 - 83.5
===================

The last time control is already fairly long time control, and at this time control LCZero v.44 seems about 1500 CCRL Elo points. For random player (0 Elo points in LCZero framework) I used my result of -150 CCRL Elo points:
http://www.talkchess.com/forum/viewtopi ... _view=flat

The transformation from LCZero framework Elo points to CCRL Elo points is close to X/2.5 - 150.
====================

The progress:
As of now, the progress does not seem to be plateauing. If it continues in the same fashion, and the framework continues its work, by the 20th of April we can expect LCZero to be about 2750 CCRL Elo points. I plotted here the progress, which doesn't show as of now plateauing, and the extrapolation to 20th of April.

Image
duncan
Posts: 12038
Joined: Mon Jul 07, 2008 10:50 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by duncan »

Laskos wrote:First, the scaling with time control.
NN with backing up the average score MCTS used in LCZero seems to scale a bit better than Alpha-Beta conventional modern engine like Zurichess. I took Zurichess Appenzeller 64-bit (Elo 1821 CCRL 40/4'), a modern conventional engine of the Zurichess line of engines, which in 2 years became pretty strong, and played 4 matches against yesterday's network 44 for LCZero, at 0.1s/move, 1s/move, 10s/move and 40s/move (a total factor of 400 in time control), and the results show a bit better scaling of LCZero:


LCZero versus Zurichess Appenzeller (Elo 1821 CCRL):
  • 0.1s / move:
    7.5 - 92.5

    1s / move:
    9.0 - 91.0

    10s / move
    11.5 - 88.5

    40s / move
    16.5 - 83.5
===================

The last time control is already fairly long time control, and at this time control LCZero v.44 seems about 1500 CCRL Elo points. For random player (0 Elo points in LCZero framework) I used my result of -150 CCRL Elo points:
http://www.talkchess.com/forum/viewtopi ... _view=flat

The transformation from LCZero framework Elo points to CCRL Elo points is close to X/2.5 - 150.
====================

The progress:
As of now, the progress does not seem to be plateauing. If it continues in the same fashion, and the framework continues its work, by the 20th of April we can expect LCZero to be about 2750 CCRL Elo points. I plotted here the progress, which doesn't show as of now plateauing, and the extrapolation to 20th of April.

Image
do you know the elo of the strongest ai chess software today ?
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Laskos »

duncan wrote:
Laskos wrote:First, the scaling with time control.
NN with backing up the average score MCTS used in LCZero seems to scale a bit better than Alpha-Beta conventional modern engine like Zurichess. I took Zurichess Appenzeller 64-bit (Elo 1821 CCRL 40/4'), a modern conventional engine of the Zurichess line of engines, which in 2 years became pretty strong, and played 4 matches against yesterday's network 44 for LCZero, at 0.1s/move, 1s/move, 10s/move and 40s/move (a total factor of 400 in time control), and the results show a bit better scaling of LCZero:


LCZero versus Zurichess Appenzeller (Elo 1821 CCRL):
  • 0.1s / move:
    7.5 - 92.5

    1s / move:
    9.0 - 91.0

    10s / move
    11.5 - 88.5

    40s / move
    16.5 - 83.5
===================

The last time control is already fairly long time control, and at this time control LCZero v.44 seems about 1500 CCRL Elo points. For random player (0 Elo points in LCZero framework) I used my result of -150 CCRL Elo points:
http://www.talkchess.com/forum/viewtopi ... _view=flat

The transformation from LCZero framework Elo points to CCRL Elo points is close to X/2.5 - 150.
====================

The progress:
As of now, the progress does not seem to be plateauing. If it continues in the same fashion, and the framework continues its work, by the 20th of April we can expect LCZero to be about 2750 CCRL Elo points. I plotted here the progress, which doesn't show as of now plateauing, and the extrapolation to 20th of April.

Image
do you know the elo of the strongest ai chess software today ?
http://www.computerchess.org.uk/ccrl/40 ... t_all.html

Look at 1 core performances, as LCZero is on one core too. Stockfish 9 on one core is about 3500 CCRL Elo.
duncan
Posts: 12038
Joined: Mon Jul 07, 2008 10:50 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by duncan »

Laskos wrote:
duncan wrote:
Laskos wrote:First, the scaling with time control.
NN with backing up the average score MCTS used in LCZero seems to scale a bit better than Alpha-Beta conventional modern engine like Zurichess. I took Zurichess Appenzeller 64-bit (Elo 1821 CCRL 40/4'), a modern conventional engine of the Zurichess line of engines, which in 2 years became pretty strong, and played 4 matches against yesterday's network 44 for LCZero, at 0.1s/move, 1s/move, 10s/move and 40s/move (a total factor of 400 in time control), and the results show a bit better scaling of LCZero:


LCZero versus Zurichess Appenzeller (Elo 1821 CCRL):
  • 0.1s / move:
    7.5 - 92.5

    1s / move:
    9.0 - 91.0

    10s / move
    11.5 - 88.5

    40s / move
    16.5 - 83.5
===================

The last time control is already fairly long time control, and at this time control LCZero v.44 seems about 1500 CCRL Elo points. For random player (0 Elo points in LCZero framework) I used my result of -150 CCRL Elo points:
http://www.talkchess.com/forum/viewtopi ... _view=flat

The transformation from LCZero framework Elo points to CCRL Elo points is close to X/2.5 - 150.
====================

The progress:
As of now, the progress does not seem to be plateauing. If it continues in the same fashion, and the framework continues its work, by the 20th of April we can expect LCZero to be about 2750 CCRL Elo points. I plotted here the progress, which doesn't show as of now plateauing, and the extrapolation to 20th of April.

Image
do you know the elo of the strongest ai chess software today ?
http://www.computerchess.org.uk/ccrl/40 ... t_all.html

Look at 1 core performances, as LCZero is on one core too. Stockfish 9 on one core is about 3500 CCRL Elo.
I meant software which uses neural networks.is it giraffe ?
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Laskos »

duncan wrote: I meant software which uses neural networks.is it giraffe ?
Giraffe, but it had a very different kind of learning and search.
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Laskos »

Now, with this post of Carlos Canavessi http://www.talkchess.com/forum/viewtopi ... 9&start=44 and the release of CPU and GPU versions, I have something odd happening. Although my GPU is weak, probably some 10 times slower than the best GTX cards, I didn't expect GPU-engine to perform so badly compared to CPU-only engine. In other GPU intensive tasks, it helps greatly over my Haswell four core i7 CPU + integrated GPU. Is LCZero using efficiently GPU?

Old LCZero shows the following:

Code: Select all

Generated 1924 moves
Detecting residual layers...v1...64 channels...6 blocks.
Initializing OpenCL.
Detected 1 OpenCL platforms.
Platform version: OpenCL 1.2 CUDA 9.1.75
Platform profile: FULL_PROFILE
Platform name:    NVIDIA CUDA
Platform vendor:  NVIDIA Corporation
Device ID:     0
Device name:   GeForce GT 730
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 388.13
Device speed:  1400 MHz
Device cores:  2 CU
Device score:  1112
Selected platform: NVIDIA CUDA
Selected device: GeForce GT 730
with OpenCL 1.2 capability.
Loaded existing SGEMM tuning.
Wavefront/Warp size: 32
Max workgroup size: 1024
Max workgroup dimensions: 1024 1024 64
BLAS Core: Haswell
New GPU LCZero shows the following (identical):

Code: Select all

Generated 1924 moves
Detecting residual layers...v1...64 channels...6 blocks.
Initializing OpenCL.
Detected 1 OpenCL platforms.
Platform version: OpenCL 1.2 CUDA 9.1.75
Platform profile: FULL_PROFILE
Platform name:    NVIDIA CUDA
Platform vendor:  NVIDIA Corporation
Device ID:     0
Device name:   GeForce GT 730
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 388.13
Device speed:  1400 MHz
Device cores:  2 CU
Device score:  1112
Selected platform: NVIDIA CUDA
Selected device: GeForce GT 730
with OpenCL 1.2 capability.
Loaded existing SGEMM tuning.
Wavefront/Warp size: 32
Max workgroup size: 1024
Max workgroup dimensions: 1024 1024 64
BLAS Core: Haswell
New CPU version shows:

Code: Select all

Generated 1924 moves
Detecting residual layers...v1...64 channels...6 blocks.
BLAS Core: Haswell


Using the old LCZero (actually the one from 1-2 days ago), and new CPU and GPU Windows binaries, I got the following:

Code: Select all

Games Completed = 300 of 300 (Avg game length = 97.535 sec)
Settings = RR/64MB/1000ms per move/M 500cp for 3 moves, D 140 moves/EPD:C:\LittleBlitzer\2moves_v1.epd(32000)
Time = 8427 sec elapsed, 0 sec remaining
 1.  LCZero Old               	53.0/200	45-139-16  	(L: m=139 t=0 i=0 a=0)	(D: r=6 i=9 f=1 s=0 a=0)	(tpm=972.7 d=11.29 nps=138)
 2.  LCZero GPU               	54.0/200	45-137-18  	(L: m=137 t=0 i=0 a=0)	(D: r=9 i=8 f=1 s=0 a=0)	(tpm=972.9 d=11.29 nps=98)
 3.  LCZero CPU               	193.0/200	190-4-6  	(L: m=4 t=0 i=0 a=0)	(D: r=3 i=3 f=0 s=0 a=0)	(tpm=960.7 d=14.66 nps=498)





   # PLAYER        : RATING  ERROR    POINTS  PLAYED     (%)   CFS(next)
   1 LCZero CPU    :  387.5   81.1     193.0     200    96.5     100    
   2 LCZero GPU    : -192.1   51.5      54.0     200    27.0      54    
   3 LCZero Old    : -195.4   52.6      53.0     200    26.5     ---    

White advantage = 0.00 +/- 28.99
Draw rate (equal opponents) = 12.69 % +/- 2.87

CPU LCZero has some 5 times higher NPS than GPU LCZero, and is much stronger. The old and the new Windows GPU version seem to perform comparably. With the old LCZero, I got

LCZero Old versus Zurichess Appenzeller (Elo 1821 CCRL):
1s/move:
9.0 - 91.0



Now, with the CPU version, I get a much better result:

LCZero CPU (one core) versus Zurichess Appenzeller (Elo 1821 CCRL):
1s/move:
33.0 - 67.0

Which gives a rating of about 1700 CCRL Elo points for CPU version. I have to revise my plot for progress and extrapolation, using stronger on my PC CPU version. From 27th to 28th of March, LCZero advanced very little, maybe it is a sign of plateauing? If not, the extrapolation on one core (for CPU version on my Haswell) would look as following:

Image
Werewolf
Posts: 1795
Joined: Thu Sep 18, 2008 10:24 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Werewolf »

When you analyse in Arena, how many rollouts / sec are you getting on with the CPU and GPU versions, please?

I'm getting 2000 - 5000 on GPU
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Laskos »

Werewolf wrote:When you analyse in Arena, how many rollouts / sec are you getting on with the CPU and GPU versions, please?

I'm getting 2000 - 5000 on GPU
Interesting. I am getting about 150 on GPU and 600 on CPU.
Werewolf
Posts: 1795
Joined: Thu Sep 18, 2008 10:24 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Werewolf »

Laskos wrote:
Werewolf wrote:When you analyse in Arena, how many rollouts / sec are you getting on with the CPU and GPU versions, please?

I'm getting 2000 - 5000 on GPU
Interesting. I am getting about 150 on GPU and 600 on CPU.

That does seem low for the GPU, though it might just be your card. The Titan V is at least 3x faster than my card which is eye opening.

By the way, do you know why there is such big regression on this site from time to time?

http://lczero.org/networks
Jhoravi
Posts: 291
Joined: Wed May 08, 2013 6:49 am

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Jhoravi »

Werewolf wrote:
By the way, do you know why there is such big regression on this site from time to time?

http://lczero.org/networks
It was caused by a bug discussed here.
https://groups.google.com/forum/#!topic ... vf9p3biLRk
Now it's fixed. :D