Lc0 v18 rc2 for 2080 Syzygy implementation

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Laskos
Posts: 8377
Joined: Wed Jul 26, 2006 8:21 pm

Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Laskos » Mon Oct 01, 2018 3:53 pm

The engine runs smoothly with new CUDA drivers.

The time control used is always 0.2s/move, Lc0 is on GTX 1060, SF8 is on 4 i7 cores.

First, now with both WDL and DTZ usage, Lc0 should convert any 6-men won positions at root with my 6-men Syzygy on SSD. Not so, on 6-men suite of hard wins at root, it missed 7 conversions out of 100. Here is the PGN, the problem seems to lie in some bug with 3-fold repetition:



Both engines had 6-men Syzygy and SF8 is known to convert all root TB positions. I didn't check the PGN thoroughly to see what exactly happens.



Second test: 6-men hard wins positions, Lc0 enabled with 5-men Syzygy against no-TBs Lc0 (both v18):

Code: Select all

Score of lc0_v18_Syzygy vs lc0_v18: 76 - 35 - 89  [0.603] 200
Elo difference: 72.25 +/- 36.09
Finished match
Not dramatically different from what I get with regular engines. One might expect that Lc0 would profit more from Syzygy because of its terrible end-play. Those are very close to root TB position, so WDL are not probed much, so third test, where WDL is probed a lot:



"Peasant Revolt" Endgame Chess
http://talkchess.com/forum3/viewtopic.p ... =0&t=65065


This starting position (and a built opening suite from it) is very sensitive to TBs, Komodo gets a boost of some 75 Elo points due to Syzygy 6-men.

Lc0 Syzygy-6 against Lc0 no-TBs

Code: Select all

Score of lc0_v18_Syzygy vs lc0_v18: 67 - 37 - 96  [0.575] 200
Elo difference: 52.51 +/- 34.82
Finished match
Not very impressive, no more improvement than regular engines get from Syzygy. Maybe because there are few hits on TBs, speed being 1000 times lower for Lc0 compared to a regular engine.

User avatar
Laskos
Posts: 8377
Joined: Wed Jul 26, 2006 8:21 pm

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Laskos » Mon Oct 01, 2018 4:30 pm

Forgot to mention, I use no adjudications at all, the games are played till the end.

can00336
Posts: 22
Joined: Sat May 16, 2015 6:07 am
Location: PA

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by can00336 » Mon Oct 01, 2018 8:47 pm

Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe

User avatar
Laskos
Posts: 8377
Joined: Wed Jul 26, 2006 8:21 pm

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Laskos » Mon Oct 01, 2018 9:16 pm

can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.

crem
Posts: 50
Joined: Wed May 23, 2018 7:29 pm

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by crem » Tue Oct 02, 2018 6:48 am

Laskos wrote:
Mon Oct 01, 2018 9:16 pm
can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.
Hi Laskos,

We have a fix for that https://github.com/LeelaChessZero/lc0/pull/410, which we have no way to check currently. Would it be possible for you to test it? The deadline for CCCC binary submission is today, and we'll have to release v0.18.1 today with a fix.
As a last resort we can push this patch blindly, but it would be nice to have it tested. :D
The CUDA build of Lc0 with this patch is https://ci.appveyor.com/api/buildjobs/3 ... %2Flc0.exe

Thanks!

User avatar
Laskos
Posts: 8377
Joined: Wed Jul 26, 2006 8:21 pm

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Laskos » Tue Oct 02, 2018 7:03 am

crem wrote:
Tue Oct 02, 2018 6:48 am
Laskos wrote:
Mon Oct 01, 2018 9:16 pm
can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.
Hi Laskos,

We have a fix for that https://github.com/LeelaChessZero/lc0/pull/410, which we have no way to check currently. Would it be possible for you to test it? The deadline for CCCC binary submission is today, and we'll have to release v0.18.1 today with a fix.
As a last resort we can push this patch blindly, but it would be nice to have it tested. :D
The CUDA build of Lc0 with this patch is https://ci.appveyor.com/api/buildjobs/3 ... %2Flc0.exe

Thanks!
In an hour or two I will post the results on 100 hard 6-men positions.
With the original v18rc2 2080 version, it misses wins due to 3-fold repetition also at much longer 2.0s/move time control, and on easy 6-men wins too, at roughly the same rate, so it's a pretty serious bug.

User avatar
Laskos
Posts: 8377
Joined: Wed Jul 26, 2006 8:21 pm

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Laskos » Tue Oct 02, 2018 8:41 am

crem wrote:
Tue Oct 02, 2018 6:48 am
Laskos wrote:
Mon Oct 01, 2018 9:16 pm
can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.
Hi Laskos,

We have a fix for that https://github.com/LeelaChessZero/lc0/pull/410, which we have no way to check currently. Would it be possible for you to test it? The deadline for CCCC binary submission is today, and we'll have to release v0.18.1 today with a fix.
As a last resort we can push this patch blindly, but it would be nice to have it tested. :D
The CUDA build of Lc0 with this patch is https://ci.appveyor.com/api/buildjobs/3 ... %2Flc0.exe

Thanks!
Sill misses by 3-fold, seems at lower rate, only 3 misses out of 100 positions. Time control for Lc0 is 0.5s/move, but as I already wrote, it seems to not matter, and even on easy 6-men wins it misses at pretty much the same rate. Here is the PGN of misses:


User avatar
George Tsavdaris
Posts: 1530
Joined: Thu Mar 09, 2006 11:35 am

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by George Tsavdaris » Tue Oct 02, 2018 9:19 am

Laskos wrote:
Tue Oct 02, 2018 7:03 am
In an hour or two I will post the results on 100 hard 6-men positions.
Can you give me or post here your 100 hard 6-men positions? :D
I know that they are from Kirill's 361 longest 6-men TBs but i want to see which ones you are using.....
After his son's birth they've asked him:
"Is it a boy or girl?"
YES! He replied.....

User avatar
Laskos
Posts: 8377
Joined: Wed Jul 26, 2006 8:21 pm

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Laskos » Tue Oct 02, 2018 9:45 am

George Tsavdaris wrote:
Tue Oct 02, 2018 9:19 am
Laskos wrote:
Tue Oct 02, 2018 7:03 am
In an hour or two I will post the results on 100 hard 6-men positions.
Can you give me or post here your 100 hard 6-men positions? :D
I know that they are from Kirill's 361 longest 6-men TBs but i want to see which ones you are using.....
Kirill longest 6-men TB wins are according to Nalimov's TBs, and many are actually draws. I will send my file of positions via PM, forgot to do it in the morning.

jkiliani
Posts: 138
Joined: Wed Jan 17, 2018 12:26 pm

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by jkiliani » Tue Oct 02, 2018 10:12 am

Laskos wrote:
Tue Oct 02, 2018 8:41 am
crem wrote:
Tue Oct 02, 2018 6:48 am
Laskos wrote:
Mon Oct 01, 2018 9:16 pm
can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.
Hi Laskos,

We have a fix for that https://github.com/LeelaChessZero/lc0/pull/410, which we have no way to check currently. Would it be possible for you to test it? The deadline for CCCC binary submission is today, and we'll have to release v0.18.1 today with a fix.
As a last resort we can push this patch blindly, but it would be nice to have it tested. :D
The CUDA build of Lc0 with this patch is https://ci.appveyor.com/api/buildjobs/3 ... %2Flc0.exe

Thanks!
Sill misses by 3-fold, seems at lower rate, only 3 misses out of 100 positions. Time control for Lc0 is 0.5s/move, but as I already wrote, it seems to not matter, and even on easy 6-men wins it misses at pretty much the same rate. Here is the PGN of misses:

Hi, thanks a lot for testing! The dev team found another bug in the tablebase implementation, where only repeats from the current position but not from all positions since the last zeroing move were considered. There is now a fix for this: https://github.com/LeelaChessZero/lc0/pull/412, with compiled binary https://ci.appveyor.com/api/buildjobs/5 ... %2Flc0.exe. If you could run your testing suite with the new build again this would be great :D Thanks again!

Post Reply