Page 1 of 4

Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Mon Oct 01, 2018 3:53 pm
by Laskos
The engine runs smoothly with new CUDA drivers.

The time control used is always 0.2s/move, Lc0 is on GTX 1060, SF8 is on 4 i7 cores.

First, now with both WDL and DTZ usage, Lc0 should convert any 6-men won positions at root with my 6-men Syzygy on SSD. Not so, on 6-men suite of hard wins at root, it missed 7 conversions out of 100. Here is the PGN, the problem seems to lie in some bug with 3-fold repetition:



Both engines had 6-men Syzygy and SF8 is known to convert all root TB positions. I didn't check the PGN thoroughly to see what exactly happens.



Second test: 6-men hard wins positions, Lc0 enabled with 5-men Syzygy against no-TBs Lc0 (both v18):

Code: Select all

Score of lc0_v18_Syzygy vs lc0_v18: 76 - 35 - 89  [0.603] 200
Elo difference: 72.25 +/- 36.09
Finished match
Not dramatically different from what I get with regular engines. One might expect that Lc0 would profit more from Syzygy because of its terrible end-play. Those are very close to root TB position, so WDL are not probed much, so third test, where WDL is probed a lot:



"Peasant Revolt" Endgame Chess
http://talkchess.com/forum3/viewtopic.p ... =0&t=65065


This starting position (and a built opening suite from it) is very sensitive to TBs, Komodo gets a boost of some 75 Elo points due to Syzygy 6-men.

Lc0 Syzygy-6 against Lc0 no-TBs

Code: Select all

Score of lc0_v18_Syzygy vs lc0_v18: 67 - 37 - 96  [0.575] 200
Elo difference: 52.51 +/- 34.82
Finished match
Not very impressive, no more improvement than regular engines get from Syzygy. Maybe because there are few hits on TBs, speed being 1000 times lower for Lc0 compared to a regular engine.

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Mon Oct 01, 2018 4:30 pm
by Laskos
Forgot to mention, I use no adjudications at all, the games are played till the end.

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Mon Oct 01, 2018 8:47 pm
by can00336
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Mon Oct 01, 2018 9:16 pm
by Laskos
can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Tue Oct 02, 2018 6:48 am
by crem
Laskos wrote:
Mon Oct 01, 2018 9:16 pm
can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.
Hi Laskos,

We have a fix for that https://github.com/LeelaChessZero/lc0/pull/410, which we have no way to check currently. Would it be possible for you to test it? The deadline for CCCC binary submission is today, and we'll have to release v0.18.1 today with a fix.
As a last resort we can push this patch blindly, but it would be nice to have it tested. :D
The CUDA build of Lc0 with this patch is https://ci.appveyor.com/api/buildjobs/3 ... %2Flc0.exe

Thanks!

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Tue Oct 02, 2018 7:03 am
by Laskos
crem wrote:
Tue Oct 02, 2018 6:48 am
Laskos wrote:
Mon Oct 01, 2018 9:16 pm
can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.
Hi Laskos,

We have a fix for that https://github.com/LeelaChessZero/lc0/pull/410, which we have no way to check currently. Would it be possible for you to test it? The deadline for CCCC binary submission is today, and we'll have to release v0.18.1 today with a fix.
As a last resort we can push this patch blindly, but it would be nice to have it tested. :D
The CUDA build of Lc0 with this patch is https://ci.appveyor.com/api/buildjobs/3 ... %2Flc0.exe

Thanks!
In an hour or two I will post the results on 100 hard 6-men positions.
With the original v18rc2 2080 version, it misses wins due to 3-fold repetition also at much longer 2.0s/move time control, and on easy 6-men wins too, at roughly the same rate, so it's a pretty serious bug.

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Tue Oct 02, 2018 8:41 am
by Laskos
crem wrote:
Tue Oct 02, 2018 6:48 am
Laskos wrote:
Mon Oct 01, 2018 9:16 pm
can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.
Hi Laskos,

We have a fix for that https://github.com/LeelaChessZero/lc0/pull/410, which we have no way to check currently. Would it be possible for you to test it? The deadline for CCCC binary submission is today, and we'll have to release v0.18.1 today with a fix.
As a last resort we can push this patch blindly, but it would be nice to have it tested. :D
The CUDA build of Lc0 with this patch is https://ci.appveyor.com/api/buildjobs/3 ... %2Flc0.exe

Thanks!
Sill misses by 3-fold, seems at lower rate, only 3 misses out of 100 positions. Time control for Lc0 is 0.5s/move, but as I already wrote, it seems to not matter, and even on easy 6-men wins it misses at pretty much the same rate. Here is the PGN of misses:


Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Tue Oct 02, 2018 9:19 am
by George Tsavdaris
Laskos wrote:
Tue Oct 02, 2018 7:03 am
In an hour or two I will post the results on 100 hard 6-men positions.
Can you give me or post here your 100 hard 6-men positions? :D
I know that they are from Kirill's 361 longest 6-men TBs but i want to see which ones you are using.....

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Tue Oct 02, 2018 9:45 am
by Laskos
George Tsavdaris wrote:
Tue Oct 02, 2018 9:19 am
Laskos wrote:
Tue Oct 02, 2018 7:03 am
In an hour or two I will post the results on 100 hard 6-men positions.
Can you give me or post here your 100 hard 6-men positions? :D
I know that they are from Kirill's 361 longest 6-men TBs but i want to see which ones you are using.....
Kirill longest 6-men TB wins are according to Nalimov's TBs, and many are actually draws. I will send my file of positions via PM, forgot to do it in the morning.

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Tue Oct 02, 2018 10:12 am
by jkiliani
Laskos wrote:
Tue Oct 02, 2018 8:41 am
crem wrote:
Tue Oct 02, 2018 6:48 am
Laskos wrote:
Mon Oct 01, 2018 9:16 pm
can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.
Hi Laskos,

We have a fix for that https://github.com/LeelaChessZero/lc0/pull/410, which we have no way to check currently. Would it be possible for you to test it? The deadline for CCCC binary submission is today, and we'll have to release v0.18.1 today with a fix.
As a last resort we can push this patch blindly, but it would be nice to have it tested. :D
The CUDA build of Lc0 with this patch is https://ci.appveyor.com/api/buildjobs/3 ... %2Flc0.exe

Thanks!
Sill misses by 3-fold, seems at lower rate, only 3 misses out of 100 positions. Time control for Lc0 is 0.5s/move, but as I already wrote, it seems to not matter, and even on easy 6-men wins it misses at pretty much the same rate. Here is the PGN of misses:

Hi, thanks a lot for testing! The dev team found another bug in the tablebase implementation, where only repeats from the current position but not from all positions since the last zeroing move were considered. There is now a fix for this: https://github.com/LeelaChessZero/lc0/pull/412, with compiled binary https://ci.appveyor.com/api/buildjobs/5 ... %2Flc0.exe. If you could run your testing suite with the new build again this would be great :D Thanks again!