I use the MEA tool.
http://rebel13.nl/rebel13/mrl.html
Direct download - http://rebel13.nl/dl.html?file=dl/mea.zip
LcZero and STS
Moderators: hgm, Rebel, chrisw
-
- Posts: 6995
- Joined: Thu Aug 18, 2011 12:04 pm
Re: LcZero and STS
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 1564
- Joined: Thu Jul 16, 2009 10:47 am
- Location: Almere, The Netherlands
Re: LcZero and STS
Strange enough network 390 performs somewhat worse compared to 395 when using LC0, probably there are algorithmic differences between LC0 and LCZero in the way the network is handled.Rebel wrote: ↑Thu Jun 14, 2018 9:43 pm390 looks obviously better, see:CMCanavessi wrote: ↑Thu Jun 14, 2018 1:06 pm Interesting results!
Would you run net 390 and/or 395 to see if there's any improvement? They are the strongest networks (in actual game play) so far.
http://rebel13.nl/zero1.html
http://rebel13.nl/zero2.html
http://rebel13.nl/zero3.html
Running 395 now.
Code: Select all
A. Processor
Brand : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz
Arch : X86_64
Count : 16
B. Engine settings
Threads : 2
Hash (mb) : 128
Time(s)/pos : 1.0
C. Test set
Filename : STS.epd
NumPos : 1500
D. Results
Engine : Rating Top1 MaxTop1 Top1Rate Score MaxScore ScoreRate
LC0-090618-395 : 2884 1081 1500 0.721 12324 15000 0.822
LC0-090618-390 : 2884 1058 1500 0.705 12194 15000 0.813
-
- Posts: 6995
- Joined: Thu Aug 18, 2011 12:04 pm
Re: LcZero and STS
I should (will) remove Fizbo from the lists, it doesn't obey the movetime command and starts moving instantly after only a couple of iterations.Joost Buijs wrote: ↑Fri Jun 15, 2018 3:33 pm I have my doubts about the accuracy of STS, most of the 1500 positions were examined with engines from several years back and it is reasonable to assume that insight has changed since then. For instance Fizbo2 scores a meagre 9027 points on this test, but in fact Fizbo2 is stronger than several other engines scoring 3000 points higher, this means that there is clearly something lacking.
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 6995
- Joined: Thu Aug 18, 2011 12:04 pm
Re: LcZero and STS
Odd indeed.....Joost Buijs wrote: ↑Fri Jun 15, 2018 3:33 pmStrange enough network 390 performs somewhat worse compared to 395 when using LC0, probably there are algorithmic differences between LC0 and LCZero in the way the network is handled.Code: Select all
A. Processor Brand : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz Arch : X86_64 Count : 16 B. Engine settings Threads : 2 Hash (mb) : 128 Time(s)/pos : 1.0 C. Test set Filename : STS.epd NumPos : 1500 D. Results Engine : Rating Top1 MaxTop1 Top1Rate Score MaxScore ScoreRate LC0-090618-395 : 2884 1081 1500 0.721 12324 15000 0.822 LC0-090618-390 : 2884 1058 1500 0.705 12194 15000 0.813
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 1142
- Joined: Thu Dec 28, 2017 4:06 pm
- Location: Argentina
Re: LcZero and STS
That's completely expected. lc0 is faster, so it produces more nodes in the same time. Hence, it's stronger.Rebel wrote: ↑Fri Jun 15, 2018 5:13 pmOdd indeed.....Joost Buijs wrote: ↑Fri Jun 15, 2018 3:33 pmStrange enough network 390 performs somewhat worse compared to 395 when using LC0, probably there are algorithmic differences between LC0 and LCZero in the way the network is handled.Code: Select all
A. Processor Brand : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz Arch : X86_64 Count : 16 B. Engine settings Threads : 2 Hash (mb) : 128 Time(s)/pos : 1.0 C. Test set Filename : STS.epd NumPos : 1500 D. Results Engine : Rating Top1 MaxTop1 Top1Rate Score MaxScore ScoreRate LC0-090618-395 : 2884 1081 1500 0.721 12324 15000 0.822 LC0-090618-390 : 2884 1058 1500 0.705 12194 15000 0.813
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls
-
- Posts: 6995
- Joined: Thu Aug 18, 2011 12:04 pm
Re: LcZero and STS
Added a tool....
With this tool you perhaps with reasonable certainty quickly can decide if a new LC-zero network weight file performs better or not. Currently version 323 has a CCRL rating of 2495 and version 390 (click on one of the 4 buttons in the upper right corner) clearly shows an improvement.
Operation
• Download LC-ZERO CPU version.
• Unzip anywhere.
• Open the folder and double-click run_mea-1s
• MEA will run the STS 1500 positions testsuite at 1 second with version 323 currently rated 2495 ELO in the CCRL 40/4 rating list and thereafter version 390. Run time ± 20-25 minutes per tested version.
• Double-click run-mrt-1s, it will open your browser and show you the results.
• That's it!
For 5 seconds testing double-click run_mea-5s and when finished double-click run_mrt-5s
For 10 seconds testing double-click run_mea-10s and when finished double-click run_mrt-10s
.....
http://rebel13.nl/rebel13/lczero.html
With this tool you perhaps with reasonable certainty quickly can decide if a new LC-zero network weight file performs better or not. Currently version 323 has a CCRL rating of 2495 and version 390 (click on one of the 4 buttons in the upper right corner) clearly shows an improvement.
Operation
• Download LC-ZERO CPU version.
• Unzip anywhere.
• Open the folder and double-click run_mea-1s
• MEA will run the STS 1500 positions testsuite at 1 second with version 323 currently rated 2495 ELO in the CCRL 40/4 rating list and thereafter version 390. Run time ± 20-25 minutes per tested version.
• Double-click run-mrt-1s, it will open your browser and show you the results.
• That's it!
For 5 seconds testing double-click run_mea-5s and when finished double-click run_mrt-5s
For 10 seconds testing double-click run_mea-10s and when finished double-click run_mrt-10s
.....
http://rebel13.nl/rebel13/lczero.html
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 1564
- Joined: Thu Jul 16, 2009 10:47 am
- Location: Almere, The Netherlands
Re: LcZero and STS
Of course LC0 is faster and (hopefully) stronger. What I said is that with LC0 network 395 performs better on STS than network 390, and with LCZero it's the other way around, this is a bit odd, don't you agree?CMCanavessi wrote: ↑Fri Jun 15, 2018 10:06 pmThat's completely expected. lc0 is faster, so it produces more nodes in the same time. Hence, it's stronger.Rebel wrote: ↑Fri Jun 15, 2018 5:13 pmOdd indeed.....Joost Buijs wrote: ↑Fri Jun 15, 2018 3:33 pmStrange enough network 390 performs somewhat worse compared to 395 when using LC0, probably there are algorithmic differences between LC0 and LCZero in the way the network is handled.Code: Select all
A. Processor Brand : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz Arch : X86_64 Count : 16 B. Engine settings Threads : 2 Hash (mb) : 128 Time(s)/pos : 1.0 C. Test set Filename : STS.epd NumPos : 1500 D. Results Engine : Rating Top1 MaxTop1 Top1Rate Score MaxScore ScoreRate LC0-090618-395 : 2884 1081 1500 0.721 12324 15000 0.822 LC0-090618-390 : 2884 1058 1500 0.705 12194 15000 0.813
-
- Posts: 1142
- Joined: Thu Dec 28, 2017 4:06 pm
- Location: Argentina
Re: LcZero and STS
Ah yes, that's a nice way to check for policy and value head. 1 net might be better for short tc, and another one may perform better with long tc.
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls