LcZero and STS

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Rebel
Posts: 4175
Joined: Thu Aug 18, 2011 10:04 am

Re: LcZero and STS

Post by Rebel » Thu Jun 14, 2018 10:23 pm

People stalk because they unconsciously love you.

Joost Buijs
Posts: 746
Joined: Thu Jul 16, 2009 8:47 am
Location: Almere, The Netherlands

Re: LcZero and STS

Post by Joost Buijs » Fri Jun 15, 2018 1:33 pm

Rebel wrote:
Thu Jun 14, 2018 7:43 pm
CMCanavessi wrote:
Thu Jun 14, 2018 11:06 am
Interesting results!

Would you run net 390 and/or 395 to see if there's any improvement? They are the strongest networks (in actual game play) so far.
390 looks obviously better, see:

http://rebel13.nl/zero1.html
http://rebel13.nl/zero2.html
http://rebel13.nl/zero3.html

Running 395 now.
Strange enough network 390 performs somewhat worse compared to 395 when using LC0, probably there are algorithmic differences between LC0 and LCZero in the way the network is handled.

Code: Select all


A. Processor
Brand          : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz
Arch           : X86_64
Count          : 16

B. Engine settings
Threads        : 2
Hash (mb)      : 128
Time(s)/pos    : 1.0

C. Test set
Filename       : STS.epd
NumPos         : 1500

D. Results
Engine                   : Rating   Top1  MaxTop1  Top1Rate  Score  MaxScore  ScoreRate
LC0-090618-395           :   2884   1081     1500     0.721  12324     15000      0.822
LC0-090618-390           :   2884   1058     1500     0.705  12194     15000      0.813

I have my doubts about the accuracy of STS, most of the 1500 positions were examined with engines from several years back and it is reasonable to assume that insight has changed since then. For instance Fizbo2 scores a meagre 9027 points on this test, but in fact Fizbo2 is stronger than several other engines scoring 3000 points higher, this means that there is clearly something lacking.

User avatar
Rebel
Posts: 4175
Joined: Thu Aug 18, 2011 10:04 am

Re: LcZero and STS

Post by Rebel » Fri Jun 15, 2018 2:16 pm

Joost Buijs wrote:
Fri Jun 15, 2018 1:33 pm
I have my doubts about the accuracy of STS, most of the 1500 positions were examined with engines from several years back and it is reasonable to assume that insight has changed since then. For instance Fizbo2 scores a meagre 9027 points on this test, but in fact Fizbo2 is stronger than several other engines scoring 3000 points higher, this means that there is clearly something lacking.
I should (will) remove Fizbo from the lists, it doesn't obey the movetime command and starts moving instantly after only a couple of iterations.
People stalk because they unconsciously love you.

User avatar
Rebel
Posts: 4175
Joined: Thu Aug 18, 2011 10:04 am

Re: LcZero and STS

Post by Rebel » Fri Jun 15, 2018 3:13 pm

Joost Buijs wrote:
Fri Jun 15, 2018 1:33 pm

Code: Select all


A. Processor
Brand          : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz
Arch           : X86_64
Count          : 16

B. Engine settings
Threads        : 2
Hash (mb)      : 128
Time(s)/pos    : 1.0

C. Test set
Filename       : STS.epd
NumPos         : 1500

D. Results
Engine                   : Rating   Top1  MaxTop1  Top1Rate  Score  MaxScore  ScoreRate
LC0-090618-395           :   2884   1081     1500     0.721  12324     15000      0.822
LC0-090618-390           :   2884   1058     1500     0.705  12194     15000      0.813

Strange enough network 390 performs somewhat worse compared to 395 when using LC0, probably there are algorithmic differences between LC0 and LCZero in the way the network is handled.
Odd indeed.....
People stalk because they unconsciously love you.

User avatar
CMCanavessi
Posts: 691
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: LcZero and STS

Post by CMCanavessi » Fri Jun 15, 2018 8:06 pm

Rebel wrote:
Fri Jun 15, 2018 3:13 pm
Joost Buijs wrote:
Fri Jun 15, 2018 1:33 pm

Code: Select all


A. Processor
Brand          : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz
Arch           : X86_64
Count          : 16

B. Engine settings
Threads        : 2
Hash (mb)      : 128
Time(s)/pos    : 1.0

C. Test set
Filename       : STS.epd
NumPos         : 1500

D. Results
Engine                   : Rating   Top1  MaxTop1  Top1Rate  Score  MaxScore  ScoreRate
LC0-090618-395           :   2884   1081     1500     0.721  12324     15000      0.822
LC0-090618-390           :   2884   1058     1500     0.705  12194     15000      0.813

Strange enough network 390 performs somewhat worse compared to 395 when using LC0, probably there are algorithmic differences between LC0 and LCZero in the way the network is handled.
Odd indeed.....
That's completely expected. lc0 is faster, so it produces more nodes in the same time. Hence, it's stronger.
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

User avatar
Rebel
Posts: 4175
Joined: Thu Aug 18, 2011 10:04 am

Re: LcZero and STS

Post by Rebel » Sat Jun 16, 2018 5:03 am

Added a tool....

With this tool you perhaps with reasonable certainty quickly can decide if a new LC-zero network weight file performs better or not. Currently version 323 has a CCRL rating of 2495 and version 390 (click on one of the 4 buttons in the upper right corner) clearly shows an improvement.

Operation
• Download LC-ZERO CPU version.
• Unzip anywhere.
• Open the folder and double-click run_mea-1s
• MEA will run the STS 1500 positions testsuite at 1 second with version 323 currently rated 2495 ELO in the CCRL 40/4 rating list and thereafter version 390. Run time ± 20-25 minutes per tested version.
• Double-click run-mrt-1s, it will open your browser and show you the results.
• That's it!

For 5 seconds testing double-click run_mea-5s and when finished double-click run_mrt-5s
For 10 seconds testing double-click run_mea-10s and when finished double-click run_mrt-10s

.....

http://rebel13.nl/rebel13/lczero.html
People stalk because they unconsciously love you.

Joost Buijs
Posts: 746
Joined: Thu Jul 16, 2009 8:47 am
Location: Almere, The Netherlands

Re: LcZero and STS

Post by Joost Buijs » Sat Jun 16, 2018 5:15 am

CMCanavessi wrote:
Fri Jun 15, 2018 8:06 pm
Rebel wrote:
Fri Jun 15, 2018 3:13 pm
Joost Buijs wrote:
Fri Jun 15, 2018 1:33 pm

Code: Select all


A. Processor
Brand          : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz
Arch           : X86_64
Count          : 16

B. Engine settings
Threads        : 2
Hash (mb)      : 128
Time(s)/pos    : 1.0

C. Test set
Filename       : STS.epd
NumPos         : 1500

D. Results
Engine                   : Rating   Top1  MaxTop1  Top1Rate  Score  MaxScore  ScoreRate
LC0-090618-395           :   2884   1081     1500     0.721  12324     15000      0.822
LC0-090618-390           :   2884   1058     1500     0.705  12194     15000      0.813

Strange enough network 390 performs somewhat worse compared to 395 when using LC0, probably there are algorithmic differences between LC0 and LCZero in the way the network is handled.
Odd indeed.....
That's completely expected. lc0 is faster, so it produces more nodes in the same time. Hence, it's stronger.
Of course LC0 is faster and (hopefully) stronger. What I said is that with LC0 network 395 performs better on STS than network 390, and with LCZero it's the other way around, this is a bit odd, don't you agree?

User avatar
CMCanavessi
Posts: 691
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: LcZero and STS

Post by CMCanavessi » Sat Jun 16, 2018 4:06 pm

Ah yes, that's a nice way to check for policy and value head. 1 net might be better for short tc, and another one may perform better with long tc.
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

Post Reply