LCZero: Progress and Scaling. Relation to CCRL Elo

[Werewolf]

Amazing. I'll wait for the training before doing tests on the new one.

[petero2]

do you know the elo of the strongest ai chess software today ?

http://www.computerchess.org.uk/ccrl/40 ... t_all.html

Look at 1 core performances, as LCZero is on one core too. Stockfish 9 on one core is about 3500 CCRL Elo.

I meant software which uses neural networks.is it giraffe ?

I think Texel Gi is the strongest that can run without a GPU/TPU. In my tests it is around 2820 on the CCRL 40/4 rating scale, so about 400 elo above Giraffe.

[Milos]

Now, with this post of Carlos Canavessi http://www.talkchess.com/forum/viewtopi ... 9&start=44 and the release of CPU and GPU versions, I have something odd happening. Although my GPU is weak, probably some 10 times slower than the best GTX cards, I didn't expect GPU-engine to perform so badly compared to CPU-only engine. In other GPU intensive tasks, it helps greatly over my Haswell four core i7 CPU + integrated GPU. Is LCZero using efficiently GPU?

Old LCZero shows the following:

Code: Select all

Generated 1924 moves
Detecting residual layers...v1...64 channels...6 blocks.
Initializing OpenCL.
Detected 1 OpenCL platforms.
Platform version: OpenCL 1.2 CUDA 9.1.75
Platform profile: FULL_PROFILE
Platform name:    NVIDIA CUDA
Platform vendor:  NVIDIA Corporation
Device ID:     0
Device name:   GeForce GT 730
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 388.13
Device speed:  1400 MHz
Device cores:  2 CU
Device score:  1112
Selected platform: NVIDIA CUDA
Selected device: GeForce GT 730
with OpenCL 1.2 capability.
Loaded existing SGEMM tuning.
Wavefront/Warp size: 32
Max workgroup size: 1024
Max workgroup dimensions: 1024 1024 64
BLAS Core: Haswell

New GPU LCZero shows the following (identical):

Code: Select all

Generated 1924 moves
Detecting residual layers...v1...64 channels...6 blocks.
Initializing OpenCL.
Detected 1 OpenCL platforms.
Platform version: OpenCL 1.2 CUDA 9.1.75
Platform profile: FULL_PROFILE
Platform name:    NVIDIA CUDA
Platform vendor:  NVIDIA Corporation
Device ID:     0
Device name:   GeForce GT 730
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 388.13
Device speed:  1400 MHz
Device cores:  2 CU
Device score:  1112
Selected platform: NVIDIA CUDA
Selected device: GeForce GT 730
with OpenCL 1.2 capability.
Loaded existing SGEMM tuning.
Wavefront/Warp size: 32
Max workgroup size: 1024
Max workgroup dimensions: 1024 1024 64
BLAS Core: Haswell

New CPU version shows:

Code: Select all

Generated 1924 moves
Detecting residual layers...v1...64 channels...6 blocks.
BLAS Core: Haswell

Using the old LCZero (actually the one from 1-2 days ago), and new CPU and GPU Windows binaries, I got the following:

Code: Select all

Games Completed = 300 of 300 (Avg game length = 97.535 sec)
Settings = RR/64MB/1000ms per move/M 500cp for 3 moves, D 140 moves/EPD:C:\LittleBlitzer\2moves_v1.epd(32000)
Time = 8427 sec elapsed, 0 sec remaining
 1.  LCZero Old               	53.0/200	45-139-16  	(L: m=139 t=0 i=0 a=0)	(D: r=6 i=9 f=1 s=0 a=0)	(tpm=972.7 d=11.29 nps=138)
 2.  LCZero GPU               	54.0/200	45-137-18  	(L: m=137 t=0 i=0 a=0)	(D: r=9 i=8 f=1 s=0 a=0)	(tpm=972.9 d=11.29 nps=98)
 3.  LCZero CPU               	193.0/200	190-4-6  	(L: m=4 t=0 i=0 a=0)	(D: r=3 i=3 f=0 s=0 a=0)	(tpm=960.7 d=14.66 nps=498)





   # PLAYER        : RATING  ERROR    POINTS  PLAYED     (%)   CFS(next)
   1 LCZero CPU    :  387.5   81.1     193.0     200    96.5     100    
   2 LCZero GPU    : -192.1   51.5      54.0     200    27.0      54    
   3 LCZero Old    : -195.4   52.6      53.0     200    26.5     ---    

White advantage = 0.00 +/- 28.99
Draw rate (equal opponents) = 12.69 % +/- 2.87

CPU LCZero has some 5 times higher NPS than GPU LCZero, and is much stronger. The old and the new Windows GPU version seem to perform comparably. With the old LCZero, I got

LCZero Old versus Zurichess Appenzeller (Elo 1821 CCRL):
1s/move:
9.0 - 91.0



Now, with the CPU version, I get a much better result:

LCZero CPU (one core) versus Zurichess Appenzeller (Elo 1821 CCRL):
1s/move:
33.0 - 67.0

Which gives a rating of about 1700 CCRL Elo points for CPU version. I have to revise my plot for progress and extrapolation, using stronger on my PC CPU version. From 27th to 28th of March, LCZero advanced very little, maybe it is a sign of plateauing? If not, the extrapolation on one core (for CPU version on my Haswell) would look as following:

This is toatally normal since your GPU is really crap (pardon my French ).
GT 730 is like 15 times slower than 1080 in terms of dot product ops, and 1080 is just slightly faster than 5960X with all 8 cores so I guess 4 cores Haswell (4770K is it?) being 5 times faster than 730 is really spot on .

[lkaufman]

do you know the elo of the strongest ai chess software today ?

http://www.computerchess.org.uk/ccrl/40 ... t_all.html

Look at 1 core performances, as LCZero is on one core too. Stockfish 9 on one core is about 3500 CCRL Elo.

I meant software which uses neural networks.is it giraffe ?

I think Texel Gi is the strongest that can run without a GPU/TPU. In my tests it is around 2820 on the CCRL 40/4 rating scale, so about 400 elo above Giraffe.

I'd like to ask a somewhat different question, what is the elo of the strongest non-private chess software (using CPU, not GPU) today that uses Monte-Carlo Tree Search, regardless of its evaluation function?

[CMCanavessi]

do you know the elo of the strongest ai chess software today ?

http://www.computerchess.org.uk/ccrl/40 ... t_all.html

Look at 1 core performances, as LCZero is on one core too. Stockfish 9 on one core is about 3500 CCRL Elo.

I meant software which uses neural networks.is it giraffe ?

I think Texel Gi is the strongest that can run without a GPU/TPU. In my tests it is around 2820 on the CCRL 40/4 rating scale, so about 400 elo above Giraffe.

I'd like to ask a somewhat different question, what is the elo of the strongest non-private chess software (using CPU, not GPU) today that uses Monte-Carlo Tree Search, regardless of its evaluation function?

Scorpio maybe?

[Werewolf]

do you know the elo of the strongest ai chess software today ?

http://www.computerchess.org.uk/ccrl/40 ... t_all.html

Look at 1 core performances, as LCZero is on one core too. Stockfish 9 on one core is about 3500 CCRL Elo.

I meant software which uses neural networks.is it giraffe ?

I think Texel Gi is the strongest that can run without a GPU/TPU. In my tests it is around 2820 on the CCRL 40/4 rating scale, so about 400 elo above Giraffe.

I'd like to ask a somewhat different question, what is the elo of the strongest non-private chess software (using CPU, not GPU) today that uses Monte-Carlo Tree Search, regardless of its evaluation function?

Scorpio maybe?

But for how long?

[Laskos]

This is toatally normal since your GPU is really crap (pardon my French ).
GT 730 is like 15 times slower than 1080 in terms of dot product ops, and 1080 is just slightly faster than 5960X with all 8 cores so I guess 4 cores Haswell (4770K is it?) being 5 times faster than 730 is really spot on .

Yes, it is so. I knew that my GPU is crap, but not that crap. I am using now the LCZero CPU engine on 4 cores, its NPS are close to a good GPU one. It performs on 4 cores remarkably well, and scales well. Against two close to 1800 CCRL Elo engines, Zurichess and Predateur, the rating and scaling are (raw data are at the end):

0.1s/move:
1600 CCRL Elo points

1s/move
1900 CCRL Elo points

10s/move
2000 CCRL Elo points

Error margins are pretty large, but the trend is obvious. So, on a good GPU, LCZero performs at about 2000 CCRL Elo points at not very short time controls, which is not bad. Maybe LTC even better.

Data:

Code: Select all

======================================================================================================================
Games Completed = 40 of 40 (Avg game length = 6.945 sec)
Settings = Gauntlet/64MB/100ms per move/M 500cp for 3 moves, D 140 moves/EPD:C:\LittleBlitzer\2moves_v1.epd(32000)
Time = 1097 sec elapsed, 0 sec remaining

 1.  LCZero CPU 4 threads        	 10.0/40	9-29-2  	(L: m=29 t=0 i=0 a=0)	(D: r=2 i=0 f=0 s=0 a=0)	(tpm=65.3 d=12.34 nps=2016)


 2.  Predateur 2.2.1 (1786)   	    15.0/20	14-4-2  	(L: m=4 t=0 i=0 a=0)	(D: r=2 i=0 f=0 s=0 a=0)	(tpm=154.3 d=11.12 nps=2975576)
 3.  Zurichess Appenzeller (1821)	 15.0/20	15-5-0  	(L: m=5 t=0 i=0 a=0)	(D: r=0 i=0 f=0 s=0 a=0)	(tpm=10.8 d=3.25 nps=1263488)
=======================================================================================================================




=======================================================================================================================
Games Completed = 40 of 40 (Avg game length = 80.504 sec)
Settings = Gauntlet/64MB/1000ms per move/M 500cp for 3 moves, D 140 moves/EPD:C:\LittleBlitzer\2moves_v1.epd(32000)
Time = 4039 sec elapsed, 0 sec remaining

 1.  LCZero CPU 4 threads        	 27.5/40	25-10-5  	(L: m=10 t=0 i=0 a=0)	(D: r=4 i=0 f=0 s=0 a=1)	(tpm=960.6 d=17.36 nps=3231)


 2.  Predateur 2.2.1 (1786)   	     8.0/20 	8-12-0  	(L: m=12 t=0 i=0 a=0)	(D: r=0 i=0 f=0 s=0 a=0)	(tpm=888.1 d=54.40 nps=3109201)
 3.  Zurichess Appenzeller (1821)	  4.5/20	 2-13-5  	(L: m=13 t=0 i=0 a=0)	(D: r=4 i=0 f=0 s=0 a=1)	(tpm=22.9 d=4.84 nps=919140)
========================================================================================================================




========================================================================================================================
Games Completed = 40 of 40 (Avg game length = 769.582 sec)
Settings = Gauntlet/64MB/10000ms per move/M 500cp for 3 moves, D 140 moves/EPD:C:\LittleBlitzer\2moves_v1.epd(32000)
Time = 31605 sec elapsed, 0 sec remaining

 1.  LCZero CPU 4 threads        	 30.5/40     29-8-3  	(L: m=7 t=0 i=0 a=1)	(D: r=2 i=0 f=0 s=0 a=1)	(tpm=9965.7 d=22.06 nps=5915)


 2.  Predateur 2.2.1 (1786)       	6.5/20      6-13-1  	(L: m=13 t=0 i=0 a=0)	(D: r=0 i=0 f=0 s=0 a=1)	(tpm=7908.0 d=35.37 nps=2870436)
 3.  Zurichess Appenzeller (1821)	 3.0/20  	 2-16-2  	(L: m=15 t=0 i=0 a=1)	(D: r=2 i=0 f=0 s=0 a=0)	(tpm=145.2 d=6.07 nps=903462)
=========================================================================================================================

[Laskos]

This is toatally normal since your GPU is really crap (pardon my French ).
GT 730 is like 15 times slower than 1080 in terms of dot product ops, and 1080 is just slightly faster than 5960X with all 8 cores so I guess 4 cores Haswell (4770K is it?) being 5 times faster than 730 is really spot on .

Yes, it is so. I knew that my GPU is crap, but not that crap. I am using now the LCZero CPU engine on 4 cores, its NPS are close to a good GPU one. It performs on 4 cores remarkably well, and scales well. Against two close to 1800 CCRL Elo engines, Zurichess and Predateur, the rating and scaling are (raw data are at the end):

0.1s/move:
1600 CCRL Elo points

1s/move
1900 CCRL Elo points

10s/move
2000 CCRL Elo points

Error margins are pretty large, but the trend is obvious. So, on a good GPU, LCZero performs at about 2000 CCRL Elo points at not very short time controls, which is not bad. Maybe LTC even better.

Data:

Code: Select all

======================================================================================================================
Games Completed = 40 of 40 (Avg game length = 6.945 sec)
Settings = Gauntlet/64MB/100ms per move/M 500cp for 3 moves, D 140 moves/EPD:C:\LittleBlitzer\2moves_v1.epd(32000)
Time = 1097 sec elapsed, 0 sec remaining

 1.  LCZero CPU 4 threads        	 10.0/40	9-29-2  	(L: m=29 t=0 i=0 a=0)	(D: r=2 i=0 f=0 s=0 a=0)	(tpm=65.3 d=12.34 nps=2016)


 2.  Predateur 2.2.1 (1786)   	    15.0/20	14-4-2  	(L: m=4 t=0 i=0 a=0)	(D: r=2 i=0 f=0 s=0 a=0)	(tpm=154.3 d=11.12 nps=2975576)
 3.  Zurichess Appenzeller (1821)	 15.0/20	15-5-0  	(L: m=5 t=0 i=0 a=0)	(D: r=0 i=0 f=0 s=0 a=0)	(tpm=10.8 d=3.25 nps=1263488)
=======================================================================================================================




=======================================================================================================================
Games Completed = 40 of 40 (Avg game length = 80.504 sec)
Settings = Gauntlet/64MB/1000ms per move/M 500cp for 3 moves, D 140 moves/EPD:C:\LittleBlitzer\2moves_v1.epd(32000)
Time = 4039 sec elapsed, 0 sec remaining

 1.  LCZero CPU 4 threads        	 27.5/40	25-10-5  	(L: m=10 t=0 i=0 a=0)	(D: r=4 i=0 f=0 s=0 a=1)	(tpm=960.6 d=17.36 nps=3231)


 2.  Predateur 2.2.1 (1786)   	     8.0/20 	8-12-0  	(L: m=12 t=0 i=0 a=0)	(D: r=0 i=0 f=0 s=0 a=0)	(tpm=888.1 d=54.40 nps=3109201)
 3.  Zurichess Appenzeller (1821)	  4.5/20	 2-13-5  	(L: m=13 t=0 i=0 a=0)	(D: r=4 i=0 f=0 s=0 a=1)	(tpm=22.9 d=4.84 nps=919140)
========================================================================================================================




========================================================================================================================
Games Completed = 40 of 40 (Avg game length = 769.582 sec)
Settings = Gauntlet/64MB/10000ms per move/M 500cp for 3 moves, D 140 moves/EPD:C:\LittleBlitzer\2moves_v1.epd(32000)
Time = 31605 sec elapsed, 0 sec remaining

 1.  LCZero CPU 4 threads        	 30.5/40     29-8-3  	(L: m=7 t=0 i=0 a=1)	(D: r=2 i=0 f=0 s=0 a=1)	(tpm=9965.7 d=22.06 nps=5915)


 2.  Predateur 2.2.1 (1786)       	6.5/20      6-13-1  	(L: m=13 t=0 i=0 a=0)	(D: r=0 i=0 f=0 s=0 a=1)	(tpm=7908.0 d=35.37 nps=2870436)
 3.  Zurichess Appenzeller (1821)	 3.0/20  	 2-16-2  	(L: m=15 t=0 i=0 a=1)	(D: r=2 i=0 f=0 s=0 a=0)	(tpm=145.2 d=6.07 nps=903462)
=========================================================================================================================

Interesting to compare LCZero with a standard engine of similar strength at 1s/move, Predateur 2.2.1.

The average length of the Wins:
LCZero: 34.7 moves
Predateur: 51.9 moves

The paired histogram of the Wins by each engine is the following:



It seems LCZero plays much better the openings and early midgames than the endgames compared to a standard engine of similar strength. Endgames will be a problem maybe for a long time.

[Jouni]

I will download LCZero when it beats SF9 and comes in one <10 MB exe-file. But I admit not to understand whole projest at all .

[CMCanavessi]

Kai, how much nps are you getting running 4 CPU threads? Would it be possible to estimate how much you would get running around 43 like TCEC does, and the approx. strenght?