LCZero: Progress and Scaling. Relation to CCRL Elo

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Werewolf
Posts: 1183
Joined: Thu Sep 18, 2008 8:24 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Werewolf » Thu Mar 29, 2018 11:16 am

Amazing. I'll wait for the training before doing tests on the new one.

petero2
Posts: 580
Joined: Mon Apr 19, 2010 5:07 pm
Location: Sweden
Contact:

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by petero2 » Fri Mar 30, 2018 9:22 am

duncan wrote:
Laskos wrote:
duncan wrote: do you know the elo of the strongest ai chess software today ?
http://www.computerchess.org.uk/ccrl/40 ... t_all.html

Look at 1 core performances, as LCZero is on one core too. Stockfish 9 on one core is about 3500 CCRL Elo.
I meant software which uses neural networks.is it giraffe ?
I think Texel Gi is the strongest that can run without a GPU/TPU. In my tests it is around 2820 on the CCRL 40/4 rating scale, so about 400 elo above Giraffe.

Milos
Posts: 3370
Joined: Wed Nov 25, 2009 12:47 am

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Milos » Fri Mar 30, 2018 5:14 pm

Laskos wrote:Now, with this post of Carlos Canavessi http://www.talkchess.com/forum/viewtopi ... 9&start=44 and the release of CPU and GPU versions, I have something odd happening. Although my GPU is weak, probably some 10 times slower than the best GTX cards, I didn't expect GPU-engine to perform so badly compared to CPU-only engine. In other GPU intensive tasks, it helps greatly over my Haswell four core i7 CPU + integrated GPU. Is LCZero using efficiently GPU?

Old LCZero shows the following:

Code: Select all

Generated 1924 moves
Detecting residual layers...v1...64 channels...6 blocks.
Initializing OpenCL.
Detected 1 OpenCL platforms.
Platform version: OpenCL 1.2 CUDA 9.1.75
Platform profile: FULL_PROFILE
Platform name:    NVIDIA CUDA
Platform vendor:  NVIDIA Corporation
Device ID:     0
Device name:   GeForce GT 730
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 388.13
Device speed:  1400 MHz
Device cores:  2 CU
Device score:  1112
Selected platform: NVIDIA CUDA
Selected device: GeForce GT 730
with OpenCL 1.2 capability.
Loaded existing SGEMM tuning.
Wavefront/Warp size: 32
Max workgroup size: 1024
Max workgroup dimensions: 1024 1024 64
BLAS Core: Haswell
New GPU LCZero shows the following (identical):

Code: Select all

Generated 1924 moves
Detecting residual layers...v1...64 channels...6 blocks.
Initializing OpenCL.
Detected 1 OpenCL platforms.
Platform version: OpenCL 1.2 CUDA 9.1.75
Platform profile: FULL_PROFILE
Platform name:    NVIDIA CUDA
Platform vendor:  NVIDIA Corporation
Device ID:     0
Device name:   GeForce GT 730
Device type:   GPU
Device vendor: NVIDIA Corporation
Device driver: 388.13
Device speed:  1400 MHz
Device cores:  2 CU
Device score:  1112
Selected platform: NVIDIA CUDA
Selected device: GeForce GT 730
with OpenCL 1.2 capability.
Loaded existing SGEMM tuning.
Wavefront/Warp size: 32
Max workgroup size: 1024
Max workgroup dimensions: 1024 1024 64
BLAS Core: Haswell
New CPU version shows:

Code: Select all

Generated 1924 moves
Detecting residual layers...v1...64 channels...6 blocks.
BLAS Core: Haswell


Using the old LCZero (actually the one from 1-2 days ago), and new CPU and GPU Windows binaries, I got the following:

Code: Select all

Games Completed = 300 of 300 (Avg game length = 97.535 sec)
Settings = RR/64MB/1000ms per move/M 500cp for 3 moves, D 140 moves/EPD:C:\LittleBlitzer\2moves_v1.epd(32000)
Time = 8427 sec elapsed, 0 sec remaining
 1.  LCZero Old               	53.0/200	45-139-16  	(L: m=139 t=0 i=0 a=0)	(D: r=6 i=9 f=1 s=0 a=0)	(tpm=972.7 d=11.29 nps=138)
 2.  LCZero GPU               	54.0/200	45-137-18  	(L: m=137 t=0 i=0 a=0)	(D: r=9 i=8 f=1 s=0 a=0)	(tpm=972.9 d=11.29 nps=98)
 3.  LCZero CPU               	193.0/200	190-4-6  	(L: m=4 t=0 i=0 a=0)	(D: r=3 i=3 f=0 s=0 a=0)	(tpm=960.7 d=14.66 nps=498)





   # PLAYER        : RATING  ERROR    POINTS  PLAYED     (%)   CFS(next)
   1 LCZero CPU    :  387.5   81.1     193.0     200    96.5     100    
   2 LCZero GPU    : -192.1   51.5      54.0     200    27.0      54    
   3 LCZero Old    : -195.4   52.6      53.0     200    26.5     ---    

White advantage = 0.00 +/- 28.99
Draw rate (equal opponents) = 12.69 % +/- 2.87

CPU LCZero has some 5 times higher NPS than GPU LCZero, and is much stronger. The old and the new Windows GPU version seem to perform comparably. With the old LCZero, I got

LCZero Old versus Zurichess Appenzeller (Elo 1821 CCRL):
1s/move:
9.0 - 91.0



Now, with the CPU version, I get a much better result:

LCZero CPU (one core) versus Zurichess Appenzeller (Elo 1821 CCRL):
1s/move:
33.0 - 67.0

Which gives a rating of about 1700 CCRL Elo points for CPU version. I have to revise my plot for progress and extrapolation, using stronger on my PC CPU version. From 27th to 28th of March, LCZero advanced very little, maybe it is a sign of plateauing? If not, the extrapolation on one core (for CPU version on my Haswell) would look as following:

Image
This is toatally normal since your GPU is really crap (pardon my French :)).
GT 730 is like 15 times slower than 1080 in terms of dot product ops, and 1080 is just slightly faster than 5960X with all 8 cores so I guess 4 cores Haswell (4770K is it?) being 5 times faster than 730 is really spot on ;).

lkaufman
Posts: 3495
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by lkaufman » Sat Mar 31, 2018 5:01 am

petero2 wrote:
duncan wrote:
Laskos wrote:
duncan wrote: do you know the elo of the strongest ai chess software today ?
http://www.computerchess.org.uk/ccrl/40 ... t_all.html

Look at 1 core performances, as LCZero is on one core too. Stockfish 9 on one core is about 3500 CCRL Elo.
I meant software which uses neural networks.is it giraffe ?
I think Texel Gi is the strongest that can run without a GPU/TPU. In my tests it is around 2820 on the CCRL 40/4 rating scale, so about 400 elo above Giraffe.
I'd like to ask a somewhat different question, what is the elo of the strongest non-private chess software (using CPU, not GPU) today that uses Monte-Carlo Tree Search, regardless of its evaluation function?
Komodo rules!

User avatar
CMCanavessi
Posts: 832
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by CMCanavessi » Sat Mar 31, 2018 5:12 am

lkaufman wrote:
petero2 wrote:
duncan wrote:
Laskos wrote:
duncan wrote: do you know the elo of the strongest ai chess software today ?
http://www.computerchess.org.uk/ccrl/40 ... t_all.html

Look at 1 core performances, as LCZero is on one core too. Stockfish 9 on one core is about 3500 CCRL Elo.
I meant software which uses neural networks.is it giraffe ?
I think Texel Gi is the strongest that can run without a GPU/TPU. In my tests it is around 2820 on the CCRL 40/4 rating scale, so about 400 elo above Giraffe.
I'd like to ask a somewhat different question, what is the elo of the strongest non-private chess software (using CPU, not GPU) today that uses Monte-Carlo Tree Search, regardless of its evaluation function?
Scorpio maybe?
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

Werewolf
Posts: 1183
Joined: Thu Sep 18, 2008 8:24 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Werewolf » Sat Mar 31, 2018 8:37 am

CMCanavessi wrote:
lkaufman wrote:
petero2 wrote:
duncan wrote:
Laskos wrote:
duncan wrote: do you know the elo of the strongest ai chess software today ?
http://www.computerchess.org.uk/ccrl/40 ... t_all.html

Look at 1 core performances, as LCZero is on one core too. Stockfish 9 on one core is about 3500 CCRL Elo.
I meant software which uses neural networks.is it giraffe ?
I think Texel Gi is the strongest that can run without a GPU/TPU. In my tests it is around 2820 on the CCRL 40/4 rating scale, so about 400 elo above Giraffe.
I'd like to ask a somewhat different question, what is the elo of the strongest non-private chess software (using CPU, not GPU) today that uses Monte-Carlo Tree Search, regardless of its evaluation function?
Scorpio maybe?
But for how long? :wink:

User avatar
Laskos
Posts: 9222
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Laskos » Sat Mar 31, 2018 10:48 am

Milos wrote: This is toatally normal since your GPU is really crap (pardon my French :)).
GT 730 is like 15 times slower than 1080 in terms of dot product ops, and 1080 is just slightly faster than 5960X with all 8 cores so I guess 4 cores Haswell (4770K is it?) being 5 times faster than 730 is really spot on ;).
Yes, it is so. I knew that my GPU is crap, but not that crap. I am using now the LCZero CPU engine on 4 cores, its NPS are close to a good GPU one. It performs on 4 cores remarkably well, and scales well. Against two close to 1800 CCRL Elo engines, Zurichess and Predateur, the rating and scaling are (raw data are at the end):

0.1s/move:
1600 CCRL Elo points

1s/move
1900 CCRL Elo points

10s/move
2000 CCRL Elo points

Error margins are pretty large, but the trend is obvious. So, on a good GPU, LCZero performs at about 2000 CCRL Elo points at not very short time controls, which is not bad. Maybe LTC even better.

Data:

Code: Select all

======================================================================================================================
Games Completed = 40 of 40 (Avg game length = 6.945 sec)
Settings = Gauntlet/64MB/100ms per move/M 500cp for 3 moves, D 140 moves/EPD:C:\LittleBlitzer\2moves_v1.epd(32000)
Time = 1097 sec elapsed, 0 sec remaining

 1.  LCZero CPU 4 threads        	 10.0/40	9-29-2  	(L: m=29 t=0 i=0 a=0)	(D: r=2 i=0 f=0 s=0 a=0)	(tpm=65.3 d=12.34 nps=2016)


 2.  Predateur 2.2.1 (1786)   	    15.0/20	14-4-2  	(L: m=4 t=0 i=0 a=0)	(D: r=2 i=0 f=0 s=0 a=0)	(tpm=154.3 d=11.12 nps=2975576)
 3.  Zurichess Appenzeller (1821)	 15.0/20	15-5-0  	(L: m=5 t=0 i=0 a=0)	(D: r=0 i=0 f=0 s=0 a=0)	(tpm=10.8 d=3.25 nps=1263488)
=======================================================================================================================




=======================================================================================================================
Games Completed = 40 of 40 (Avg game length = 80.504 sec)
Settings = Gauntlet/64MB/1000ms per move/M 500cp for 3 moves, D 140 moves/EPD:C:\LittleBlitzer\2moves_v1.epd(32000)
Time = 4039 sec elapsed, 0 sec remaining

 1.  LCZero CPU 4 threads        	 27.5/40	25-10-5  	(L: m=10 t=0 i=0 a=0)	(D: r=4 i=0 f=0 s=0 a=1)	(tpm=960.6 d=17.36 nps=3231)


 2.  Predateur 2.2.1 (1786)   	     8.0/20 	8-12-0  	(L: m=12 t=0 i=0 a=0)	(D: r=0 i=0 f=0 s=0 a=0)	(tpm=888.1 d=54.40 nps=3109201)
 3.  Zurichess Appenzeller (1821)	  4.5/20	 2-13-5  	(L: m=13 t=0 i=0 a=0)	(D: r=4 i=0 f=0 s=0 a=1)	(tpm=22.9 d=4.84 nps=919140)
========================================================================================================================




========================================================================================================================
Games Completed = 40 of 40 (Avg game length = 769.582 sec)
Settings = Gauntlet/64MB/10000ms per move/M 500cp for 3 moves, D 140 moves/EPD:C:\LittleBlitzer\2moves_v1.epd(32000)
Time = 31605 sec elapsed, 0 sec remaining

 1.  LCZero CPU 4 threads        	 30.5/40     29-8-3  	(L: m=7 t=0 i=0 a=1)	(D: r=2 i=0 f=0 s=0 a=1)	(tpm=9965.7 d=22.06 nps=5915)


 2.  Predateur 2.2.1 (1786)       	6.5/20      6-13-1  	(L: m=13 t=0 i=0 a=0)	(D: r=0 i=0 f=0 s=0 a=1)	(tpm=7908.0 d=35.37 nps=2870436)
 3.  Zurichess Appenzeller (1821)	 3.0/20  	 2-16-2  	(L: m=15 t=0 i=0 a=1)	(D: r=2 i=0 f=0 s=0 a=0)	(tpm=145.2 d=6.07 nps=903462)
=========================================================================================================================

User avatar
Laskos
Posts: 9222
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Laskos » Sat Mar 31, 2018 12:25 pm

Laskos wrote:
Milos wrote: This is toatally normal since your GPU is really crap (pardon my French :)).
GT 730 is like 15 times slower than 1080 in terms of dot product ops, and 1080 is just slightly faster than 5960X with all 8 cores so I guess 4 cores Haswell (4770K is it?) being 5 times faster than 730 is really spot on ;).
Yes, it is so. I knew that my GPU is crap, but not that crap. I am using now the LCZero CPU engine on 4 cores, its NPS are close to a good GPU one. It performs on 4 cores remarkably well, and scales well. Against two close to 1800 CCRL Elo engines, Zurichess and Predateur, the rating and scaling are (raw data are at the end):

0.1s/move:
1600 CCRL Elo points

1s/move
1900 CCRL Elo points

10s/move
2000 CCRL Elo points

Error margins are pretty large, but the trend is obvious. So, on a good GPU, LCZero performs at about 2000 CCRL Elo points at not very short time controls, which is not bad. Maybe LTC even better.

Data:

Code: Select all

======================================================================================================================
Games Completed = 40 of 40 (Avg game length = 6.945 sec)
Settings = Gauntlet/64MB/100ms per move/M 500cp for 3 moves, D 140 moves/EPD:C:\LittleBlitzer\2moves_v1.epd(32000)
Time = 1097 sec elapsed, 0 sec remaining

 1.  LCZero CPU 4 threads        	 10.0/40	9-29-2  	(L: m=29 t=0 i=0 a=0)	(D: r=2 i=0 f=0 s=0 a=0)	(tpm=65.3 d=12.34 nps=2016)


 2.  Predateur 2.2.1 (1786)   	    15.0/20	14-4-2  	(L: m=4 t=0 i=0 a=0)	(D: r=2 i=0 f=0 s=0 a=0)	(tpm=154.3 d=11.12 nps=2975576)
 3.  Zurichess Appenzeller (1821)	 15.0/20	15-5-0  	(L: m=5 t=0 i=0 a=0)	(D: r=0 i=0 f=0 s=0 a=0)	(tpm=10.8 d=3.25 nps=1263488)
=======================================================================================================================




=======================================================================================================================
Games Completed = 40 of 40 (Avg game length = 80.504 sec)
Settings = Gauntlet/64MB/1000ms per move/M 500cp for 3 moves, D 140 moves/EPD:C:\LittleBlitzer\2moves_v1.epd(32000)
Time = 4039 sec elapsed, 0 sec remaining

 1.  LCZero CPU 4 threads        	 27.5/40	25-10-5  	(L: m=10 t=0 i=0 a=0)	(D: r=4 i=0 f=0 s=0 a=1)	(tpm=960.6 d=17.36 nps=3231)


 2.  Predateur 2.2.1 (1786)   	     8.0/20 	8-12-0  	(L: m=12 t=0 i=0 a=0)	(D: r=0 i=0 f=0 s=0 a=0)	(tpm=888.1 d=54.40 nps=3109201)
 3.  Zurichess Appenzeller (1821)	  4.5/20	 2-13-5  	(L: m=13 t=0 i=0 a=0)	(D: r=4 i=0 f=0 s=0 a=1)	(tpm=22.9 d=4.84 nps=919140)
========================================================================================================================




========================================================================================================================
Games Completed = 40 of 40 (Avg game length = 769.582 sec)
Settings = Gauntlet/64MB/10000ms per move/M 500cp for 3 moves, D 140 moves/EPD:C:\LittleBlitzer\2moves_v1.epd(32000)
Time = 31605 sec elapsed, 0 sec remaining

 1.  LCZero CPU 4 threads        	 30.5/40     29-8-3  	(L: m=7 t=0 i=0 a=1)	(D: r=2 i=0 f=0 s=0 a=1)	(tpm=9965.7 d=22.06 nps=5915)


 2.  Predateur 2.2.1 (1786)       	6.5/20      6-13-1  	(L: m=13 t=0 i=0 a=0)	(D: r=0 i=0 f=0 s=0 a=1)	(tpm=7908.0 d=35.37 nps=2870436)
 3.  Zurichess Appenzeller (1821)	 3.0/20  	 2-16-2  	(L: m=15 t=0 i=0 a=1)	(D: r=2 i=0 f=0 s=0 a=0)	(tpm=145.2 d=6.07 nps=903462)
=========================================================================================================================
Interesting to compare LCZero with a standard engine of similar strength at 1s/move, Predateur 2.2.1.

The average length of the Wins:
LCZero: 34.7 moves
Predateur: 51.9 moves

The paired histogram of the Wins by each engine is the following:

Image

It seems LCZero plays much better the openings and early midgames than the endgames compared to a standard engine of similar strength. Endgames will be a problem maybe for a long time.

Jouni
Posts: 1953
Joined: Wed Mar 08, 2006 7:15 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Jouni » Sat Mar 31, 2018 2:51 pm

I will download LCZero when it beats SF9 and comes in one <10 MB exe-file. But I admit not to understand whole projest at all :) .
Jouni

User avatar
CMCanavessi
Posts: 832
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by CMCanavessi » Sat Mar 31, 2018 4:49 pm

Kai, how much nps are you getting running 4 CPU threads? Would it be possible to estimate how much you would get running around 43 like TCEC does, and the approx. strenght?
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

Post Reply