Page 2 of 4

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Tue Oct 02, 2018 10:16 am
by Laskos
jkiliani wrote:
Tue Oct 02, 2018 10:12 am
Laskos wrote:
Tue Oct 02, 2018 8:41 am
crem wrote:
Tue Oct 02, 2018 6:48 am
Laskos wrote:
Mon Oct 01, 2018 9:16 pm
can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.
Hi Laskos,

We have a fix for that https://github.com/LeelaChessZero/lc0/pull/410, which we have no way to check currently. Would it be possible for you to test it? The deadline for CCCC binary submission is today, and we'll have to release v0.18.1 today with a fix.
As a last resort we can push this patch blindly, but it would be nice to have it tested. :D
The CUDA build of Lc0 with this patch is https://ci.appveyor.com/api/buildjobs/3 ... %2Flc0.exe

Thanks!
Sill misses by 3-fold, seems at lower rate, only 3 misses out of 100 positions. Time control for Lc0 is 0.5s/move, but as I already wrote, it seems to not matter, and even on easy 6-men wins it misses at pretty much the same rate. Here is the PGN of misses:

Hi, thanks a lot for testing! The dev team found another bug in the tablebase implementation, where only repeats from the current position but not from all positions since the last zeroing move were considered. There is now a fix for this: https://github.com/LeelaChessZero/lc0/pull/412, with compiled binary https://ci.appveyor.com/api/buildjobs/5 ... %2Flc0.exe. If you could run your testing suite with the new build again this would be great :D Thanks again!
The result on 100 positions in an hour or so. At short time control, as it seems to not matter.

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Tue Oct 02, 2018 11:09 am
by Laskos
Laskos wrote:
Tue Oct 02, 2018 10:16 am
jkiliani wrote:
Tue Oct 02, 2018 10:12 am
Laskos wrote:
Tue Oct 02, 2018 8:41 am
crem wrote:
Tue Oct 02, 2018 6:48 am
Laskos wrote:
Mon Oct 01, 2018 9:16 pm
can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.
Hi Laskos,

We have a fix for that https://github.com/LeelaChessZero/lc0/pull/410, which we have no way to check currently. Would it be possible for you to test it? The deadline for CCCC binary submission is today, and we'll have to release v0.18.1 today with a fix.
As a last resort we can push this patch blindly, but it would be nice to have it tested. :D
The CUDA build of Lc0 with this patch is https://ci.appveyor.com/api/buildjobs/3 ... %2Flc0.exe

Thanks!
Sill misses by 3-fold, seems at lower rate, only 3 misses out of 100 positions. Time control for Lc0 is 0.5s/move, but as I already wrote, it seems to not matter, and even on easy 6-men wins it misses at pretty much the same rate. Here is the PGN of misses:

Hi, thanks a lot for testing! The dev team found another bug in the tablebase implementation, where only repeats from the current position but not from all positions since the last zeroing move were considered. There is now a fix for this: https://github.com/LeelaChessZero/lc0/pull/412, with compiled binary https://ci.appveyor.com/api/buildjobs/5 ... %2Flc0.exe. If you could run your testing suite with the new build again this would be great :D Thanks again!
The result on 100 positions in an hour or so. At short time control, as it seems to not matter.
No misses this time on 100 hard 6-men wins.
Maybe I will check on say 200 other 6-men won positions, but it seems fine now.

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Tue Oct 02, 2018 11:12 am
by jkiliani
Laskos wrote:
Tue Oct 02, 2018 11:09 am
No misses this time on 100 hard 6-men wins.
Maybe I will check on say 200 other 6-men won positions, but it seems fine now.
Great, thank you for the whole Leela Dev team! Do you have any testing suites about holding difficult draws, or avoiding losses on TB lost positions against opponents without TB?

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Tue Oct 02, 2018 11:22 am
by Laskos
jkiliani wrote:
Tue Oct 02, 2018 11:12 am
Laskos wrote:
Tue Oct 02, 2018 11:09 am
No misses this time on 100 hard 6-men wins.
Maybe I will check on say 200 other 6-men won positions, but it seems fine now.
Great, thank you! Do you have any testing suites about holding difficult draws, or avoiding losses on TB lost positions against opponents without TB?
I tested yesterday on 100 long 6-men draws (not sure if they are exactly hard, but many of them are), no losses. Avoiding TB losses against non-TB engine needs comparison with another Syzygy-enabled engine against that non-TB engine. Might compare Lc0_Syzygy against SF8_no_TB compared to SF8_Syzygy against SF8_no_TB on 6-men hard wins. Will depend on time control used too.

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Tue Oct 02, 2018 11:53 am
by Javier Ros
Very interesting experiment and great work! :D

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Tue Oct 02, 2018 12:33 pm
by Laskos
Laskos wrote:
Tue Oct 02, 2018 11:22 am
jkiliani wrote:
Tue Oct 02, 2018 11:12 am
Laskos wrote:
Tue Oct 02, 2018 11:09 am
No misses this time on 100 hard 6-men wins.
Maybe I will check on say 200 other 6-men won positions, but it seems fine now.
Great, thank you! Do you have any testing suites about holding difficult draws, or avoiding losses on TB lost positions against opponents without TB?
I tested yesterday on 100 long 6-men draws (not sure if they are exactly hard, but many of them are), no losses. Avoiding TB losses against non-TB engine needs comparison with another Syzygy-enabled engine against that non-TB engine. Might compare Lc0_Syzygy against SF8_no_TB compared to SF8_Syzygy against SF8_no_TB on 6-men hard wins. Will depend on time control used too.
Out of 50 6-men TB losses, Lc0_Syzygy (the fixed one) saves 13 against SF8_no_TB, SF8_Syzygy saves 15 against SF8_no_TB. So, behaves similarly to SF8, which has a good TB implementation (and good endgame eval, which Lc0 doesn't). Might check how many saves SF dev.

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Tue Oct 02, 2018 1:03 pm
by Damir
jkiliani wrote:
Tue Oct 02, 2018 10:12 am
Laskos wrote:
Tue Oct 02, 2018 8:41 am
crem wrote:
Tue Oct 02, 2018 6:48 am
Laskos wrote:
Mon Oct 01, 2018 9:16 pm
can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.
Hi Laskos,

We have a fix for that https://github.com/LeelaChessZero/lc0/pull/410, which we have no way to check currently. Would it be possible for you to test it? The deadline for CCCC binary submission is today, and we'll have to release v0.18.1 today with a fix.
As a last resort we can push this patch blindly, but it would be nice to have it tested. :D
The CUDA build of Lc0 with this patch is https://ci.appveyor.com/api/buildjobs/3 ... %2Flc0.exe

Thanks!
Sill misses by 3-fold, seems at lower rate, only 3 misses out of 100 positions. Time control for Lc0 is 0.5s/move, but as I already wrote, it seems to not matter, and even on easy 6-men wins it misses at pretty much the same rate. Here is the PGN of misses:

Hi, thanks a lot for testing! The dev team found another bug in the tablebase implementation, where only repeats from the current position but not from all positions since the last zeroing move were considered. There is now a fix for this: https://github.com/LeelaChessZero/lc0/pull/412, with compiled binary https://ci.appveyor.com/api/buildjobs/5 ... %2Flc0.exe. If you could run your testing suite with the new build again this would be great :D Thanks again!
How about fixing a bug where engine will suddenly resign during the game out of nowhere? This happens during draw or won game, and especially in long games...

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Tue Oct 02, 2018 1:06 pm
by jkiliani
Damir wrote:
Tue Oct 02, 2018 1:03 pm
How about fixing a bug where engine will suddenly resign during the game out of nowhere? This happens during draw or won game, and especially in long games...
To fix bugs, we need to first reproduce them. Can you give a .pgn file where this happened for you, as well as specifics about the binary you used and your command line parameters?

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Tue Oct 02, 2018 1:33 pm
by Laskos
Laskos wrote:
Tue Oct 02, 2018 12:33 pm
Laskos wrote:
Tue Oct 02, 2018 11:22 am
jkiliani wrote:
Tue Oct 02, 2018 11:12 am
Laskos wrote:
Tue Oct 02, 2018 11:09 am
No misses this time on 100 hard 6-men wins.
Maybe I will check on say 200 other 6-men won positions, but it seems fine now.
Great, thank you! Do you have any testing suites about holding difficult draws, or avoiding losses on TB lost positions against opponents without TB?
I tested yesterday on 100 long 6-men draws (not sure if they are exactly hard, but many of them are), no losses. Avoiding TB losses against non-TB engine needs comparison with another Syzygy-enabled engine against that non-TB engine. Might compare Lc0_Syzygy against SF8_no_TB compared to SF8_Syzygy against SF8_no_TB on 6-men hard wins. Will depend on time control used too.
Out of 50 6-men TB losses, Lc0_Syzygy (the fixed one) saves 13 against SF8_no_TB, SF8_Syzygy saves 15 against SF8_no_TB. So, behaves similarly to SF8, which has a good TB implementation (and good endgame eval, which Lc0 doesn't). Might check how many saves SF dev.
SF_dev_Syzygy saves 23 out of 50 TB losses. Might be due to different Syzygy implementation, stronger play, etc., I don't know.

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Posted: Tue Oct 02, 2018 1:36 pm
by nabildanial
Damir wrote:
Tue Oct 02, 2018 1:03 pm
jkiliani wrote:
Tue Oct 02, 2018 10:12 am
Laskos wrote:
Tue Oct 02, 2018 8:41 am
crem wrote:
Tue Oct 02, 2018 6:48 am
Laskos wrote:
Mon Oct 01, 2018 9:16 pm
can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.
Hi Laskos,

We have a fix for that https://github.com/LeelaChessZero/lc0/pull/410, which we have no way to check currently. Would it be possible for you to test it? The deadline for CCCC binary submission is today, and we'll have to release v0.18.1 today with a fix.
As a last resort we can push this patch blindly, but it would be nice to have it tested. :D
The CUDA build of Lc0 with this patch is https://ci.appveyor.com/api/buildjobs/3 ... %2Flc0.exe

Thanks!
Sill misses by 3-fold, seems at lower rate, only 3 misses out of 100 positions. Time control for Lc0 is 0.5s/move, but as I already wrote, it seems to not matter, and even on easy 6-men wins it misses at pretty much the same rate. Here is the PGN of misses:

Hi, thanks a lot for testing! The dev team found another bug in the tablebase implementation, where only repeats from the current position but not from all positions since the last zeroing move were considered. There is now a fix for this: https://github.com/LeelaChessZero/lc0/pull/412, with compiled binary https://ci.appveyor.com/api/buildjobs/5 ... %2Flc0.exe. If you could run your testing suite with the new build again this would be great :D Thanks again!
How about fixing a bug where engine will suddenly resign during the game out of nowhere? This happens during draw or won game, and especially in long games...
How does a UCI engine resign in games?