Lc0 v18 rc2 for 2080 Syzygy implementation

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Damir
Posts: 1921
Joined: Mon Feb 11, 2008 2:53 pm

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Damir » Tue Oct 02, 2018 1:48 pm

This happens when playing on Playchess server... I don't know why, but it happens… Maybe Leela has bug located in its time code when playing 16+0, or much longer games like 60/15 , which cause it to suddenly resign the game before it is even finished? :( :(

jkiliani
Posts: 138
Joined: Wed Jan 17, 2018 12:26 pm

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by jkiliani » Tue Oct 02, 2018 3:11 pm

Laskos wrote:
Tue Oct 02, 2018 12:33 pm
I tested yesterday on 100 long 6-men draws (not sure if they are exactly hard, but many of them are), no losses. Avoiding TB losses against non-TB engine needs comparison with another Syzygy-enabled engine against that non-TB engine. Might compare Lc0_Syzygy against SF8_no_TB compared to SF8_Syzygy against SF8_no_TB on 6-men hard wins. Will depend on time control used too.

Out of 50 6-men TB losses, Lc0_Syzygy (the fixed one) saves 13 against SF8_no_TB, SF8_Syzygy saves 15 against SF8_no_TB. So, behaves similarly to SF8, which has a good TB implementation (and good endgame eval, which Lc0 doesn't). Might check how many saves SF dev.

SF_dev_Syzygy saves 23 out of 50 TB losses. Might be due to different Syzygy implementation, stronger play, etc., I don't know.
Nice experiment, thanks! If Leela can play TB wins and draws perfectly and avoid some TB losses against no-TB opponents, the implementation can probably be considered a success. All engines strong enough for a reasonable chance to reach a TB win against Leela have TB themselves, so the loss avoidance likely isn't that important.

In a few months when we have strong Leela networks based on TB rescoring, it will be interesting how those will score in these benchmarks without using TB themselves...

User avatar
Laskos
Posts: 8488
Joined: Wed Jul 26, 2006 8:21 pm

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Laskos » Tue Oct 02, 2018 6:23 pm

jkiliani wrote:
Tue Oct 02, 2018 3:11 pm
Laskos wrote:
Tue Oct 02, 2018 12:33 pm
I tested yesterday on 100 long 6-men draws (not sure if they are exactly hard, but many of them are), no losses. Avoiding TB losses against non-TB engine needs comparison with another Syzygy-enabled engine against that non-TB engine. Might compare Lc0_Syzygy against SF8_no_TB compared to SF8_Syzygy against SF8_no_TB on 6-men hard wins. Will depend on time control used too.

Out of 50 6-men TB losses, Lc0_Syzygy (the fixed one) saves 13 against SF8_no_TB, SF8_Syzygy saves 15 against SF8_no_TB. So, behaves similarly to SF8, which has a good TB implementation (and good endgame eval, which Lc0 doesn't). Might check how many saves SF dev.

SF_dev_Syzygy saves 23 out of 50 TB losses. Might be due to different Syzygy implementation, stronger play, etc., I don't know.
Nice experiment, thanks! If Leela can play TB wins and draws perfectly and avoid some TB losses against no-TB opponents, the implementation can probably be considered a success. All engines strong enough for a reasonable chance to reach a TB win against Leela have TB themselves, so the loss avoidance likely isn't that important.

In a few months when we have strong Leela networks based on TB rescoring, it will be interesting how those will score in these benchmarks without using TB themselves...
I checked for another set of 400 won 6-men positions against SF8_Syzygy, not a single miss.
TBs are only up to 7 men, Leela often misevaluates heavily earlier endgames with many more pieces, I am bit skeptical that based on TB rescoring, Leela will become comparable to top engines in endgames, with their hand-crafted knowledge, which is often general and pretty abstract.

Milos
Posts: 3243
Joined: Wed Nov 25, 2009 12:47 am

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Milos » Tue Oct 02, 2018 7:26 pm

Laskos wrote:
Tue Oct 02, 2018 6:23 pm
I checked for another set of 400 won 6-men positions against SF8_Syzygy, not a single miss.
TBs are only up to 7 men, Leela often misevaluates heavily earlier endgames with many more pieces, I am bit skeptical that based on TB rescoring, Leela will become comparable to top engines in endgames, with their hand-crafted knowledge, which is often general and pretty abstract.
You scepticism is totally in place. 6-men at root solves almost nothing for Lc0.
Problem is adding 6-men properly to leaves is also not working due to MCTS using averaging as an operator. There is no easy solution. Maybe some hybrid approach that would switch operator to minmax once you start reaching TB positions at leaves, but that is not easy to implement and even harder to tune.

chrisw
Posts: 781
Joined: Tue Apr 03, 2012 2:28 pm

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by chrisw » Tue Oct 02, 2018 7:58 pm

Milos wrote:
Tue Oct 02, 2018 7:26 pm
Laskos wrote:
Tue Oct 02, 2018 6:23 pm
I checked for another set of 400 won 6-men positions against SF8_Syzygy, not a single miss.
TBs are only up to 7 men, Leela often misevaluates heavily earlier endgames with many more pieces, I am bit skeptical that based on TB rescoring, Leela will become comparable to top engines in endgames, with their hand-crafted knowledge, which is often general and pretty abstract.
You scepticism is totally in place. 6-men at root solves almost nothing for Lc0.
Problem is adding 6-men properly to leaves is also not working due to MCTS using averaging as an operator. There is no easy solution. Maybe some hybrid approach that would switch operator to minmax once you start reaching TB positions at leaves, but that is not easy to implement and even harder to tune.
it's a relatively simple modification to the backed up value. if you ask for a value at a leaf, and the value is an absolute value (eg comes from the egtb or actually any terminal position) then:
if win, don't average it in, but back up the absolute win value
if draw, back up the max(0.5, average so far)
if loss, back up the average so far

you can get extra clever and flag terminal nodes, then test status from the ply below. If there's one terminal win above then this ply is a terminal loss in all cases. if all terminal losses above, then this ply is a terminal win. and so on.
Chris Whittington
Ontological Terrorist
Lead Philosopher Computer Chess
Comrade Commissar Aspergers Liberation Front
Ostranenye Catalyst
Zero tolerance personal comments
5-4-8 tritype, INTP, BCF 221
facebook: chris.whittington.522
twitter: Skool_Dog

Milos
Posts: 3243
Joined: Wed Nov 25, 2009 12:47 am

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Milos » Tue Oct 02, 2018 8:07 pm

chrisw wrote:
Tue Oct 02, 2018 7:58 pm
Milos wrote:
Tue Oct 02, 2018 7:26 pm
Laskos wrote:
Tue Oct 02, 2018 6:23 pm
I checked for another set of 400 won 6-men positions against SF8_Syzygy, not a single miss.
TBs are only up to 7 men, Leela often misevaluates heavily earlier endgames with many more pieces, I am bit skeptical that based on TB rescoring, Leela will become comparable to top engines in endgames, with their hand-crafted knowledge, which is often general and pretty abstract.
You scepticism is totally in place. 6-men at root solves almost nothing for Lc0.
Problem is adding 6-men properly to leaves is also not working due to MCTS using averaging as an operator. There is no easy solution. Maybe some hybrid approach that would switch operator to minmax once you start reaching TB positions at leaves, but that is not easy to implement and even harder to tune.
it's a relatively simple modification to the backed up value. if you ask for a value at a leaf, and the value is an absolute value (eg comes from the egtb or actually any terminal position) then:
if win, don't average it in, but back up the absolute win value
if draw, back up the max(0.5, average so far)
if loss, back up the average so far

you can get extra clever and flag terminal nodes, then test status from the ply below. If there's one terminal win above then this ply is a terminal loss in all cases. if all terminal losses above, then this ply is a terminal win. and so on.
Again, not so easy. You backup lets say a win, move one ply back to the root and do what, since that becomes the worst move for your opponent you just average it up and loose any kind of information. It doesn't propagate further up the tree.
The only thing that is actually easy to implement and might be able to work is to give higher weight to terminal positions when averaging. But that would require a lot of tuning.

chrisw
Posts: 781
Joined: Tue Apr 03, 2012 2:28 pm

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by chrisw » Tue Oct 02, 2018 8:38 pm

Milos wrote:
Tue Oct 02, 2018 8:07 pm
chrisw wrote:
Tue Oct 02, 2018 7:58 pm
Milos wrote:
Tue Oct 02, 2018 7:26 pm
Laskos wrote:
Tue Oct 02, 2018 6:23 pm
I checked for another set of 400 won 6-men positions against SF8_Syzygy, not a single miss.
TBs are only up to 7 men, Leela often misevaluates heavily earlier endgames with many more pieces, I am bit skeptical that based on TB rescoring, Leela will become comparable to top engines in endgames, with their hand-crafted knowledge, which is often general and pretty abstract.
You scepticism is totally in place. 6-men at root solves almost nothing for Lc0.
Problem is adding 6-men properly to leaves is also not working due to MCTS using averaging as an operator. There is no easy solution. Maybe some hybrid approach that would switch operator to minmax once you start reaching TB positions at leaves, but that is not easy to implement and even harder to tune.
it's a relatively simple modification to the backed up value. if you ask for a value at a leaf, and the value is an absolute value (eg comes from the egtb or actually any terminal position) then:
if win, don't average it in, but back up the absolute win value
if draw, back up the max(0.5, average so far)
if loss, back up the average so far

you can get extra clever and flag terminal nodes, then test status from the ply below. If there's one terminal win above then this ply is a terminal loss in all cases. if all terminal losses above, then this ply is a terminal win. and so on.
Again, not so easy. You backup lets say a win, move one ply back to the root and do what, since that becomes the worst move for your opponent you just average it up and loose any kind of information. It doesn't propagate further up the tree.
it is a bit of a hassle to get it to work, you are kind of minimaxing, but only in terminal situations, and you have to treat win nodes differently to loss nodes.
a win node backs up to a "terminal" loss the ply below. end of story. (and an immediate algorithmic gain over averaging)
a loss node only backs up to a "terminal" win one ply below if all siblings are lost. so you need to keep some sort of flagging count.
Think about it, you can make it work.
Chris Whittington
Ontological Terrorist
Lead Philosopher Computer Chess
Comrade Commissar Aspergers Liberation Front
Ostranenye Catalyst
Zero tolerance personal comments
5-4-8 tritype, INTP, BCF 221
facebook: chris.whittington.522
twitter: Skool_Dog

User avatar
Laskos
Posts: 8488
Joined: Wed Jul 26, 2006 8:21 pm

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Laskos » Wed Oct 03, 2018 7:16 am

jkiliani wrote:
Tue Oct 02, 2018 3:11 pm
Laskos wrote:
Tue Oct 02, 2018 12:33 pm
I tested yesterday on 100 long 6-men draws (not sure if they are exactly hard, but many of them are), no losses. Avoiding TB losses against non-TB engine needs comparison with another Syzygy-enabled engine against that non-TB engine. Might compare Lc0_Syzygy against SF8_no_TB compared to SF8_Syzygy against SF8_no_TB on 6-men hard wins. Will depend on time control used too.

Out of 50 6-men TB losses, Lc0_Syzygy (the fixed one) saves 13 against SF8_no_TB, SF8_Syzygy saves 15 against SF8_no_TB. So, behaves similarly to SF8, which has a good TB implementation (and good endgame eval, which Lc0 doesn't). Might check how many saves SF dev.

SF_dev_Syzygy saves 23 out of 50 TB losses. Might be due to different Syzygy implementation, stronger play, etc., I don't know.
Nice experiment, thanks! If Leela can play TB wins and draws perfectly and avoid some TB losses against no-TB opponents, the implementation can probably be considered a success. All engines strong enough for a reasonable chance to reach a TB win against Leela have TB themselves, so the loss avoidance likely isn't that important.
Yes, the implementation is fine and does help a lot, especially the DTZ part. Just compare to what Lc0 does without Syzygy on 6-men TB positions at root:

Out of 50 6-men TB wins, Lc0_no_TBs against SF_dev_Syzygy converts 6, misses as draws 38, and misses even as losses 6. So, from 6-men won positions, Lc0_no_TB scores 50% against SF_dev_Syzygy from 6-men wins!

So, the Syzygy implementation is fine and helps a lot in this case, as Lc0 plays really badly in these late endgames.

Joerg Oster
Posts: 641
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Joerg Oster » Wed Oct 03, 2018 8:39 am

chrisw wrote:
Tue Oct 02, 2018 7:58 pm
Milos wrote:
Tue Oct 02, 2018 7:26 pm
Laskos wrote:
Tue Oct 02, 2018 6:23 pm
I checked for another set of 400 won 6-men positions against SF8_Syzygy, not a single miss.
TBs are only up to 7 men, Leela often misevaluates heavily earlier endgames with many more pieces, I am bit skeptical that based on TB rescoring, Leela will become comparable to top engines in endgames, with their hand-crafted knowledge, which is often general and pretty abstract.
You scepticism is totally in place. 6-men at root solves almost nothing for Lc0.
Problem is adding 6-men properly to leaves is also not working due to MCTS using averaging as an operator. There is no easy solution. Maybe some hybrid approach that would switch operator to minmax once you start reaching TB positions at leaves, but that is not easy to implement and even harder to tune.
it's a relatively simple modification to the backed up value. if you ask for a value at a leaf, and the value is an absolute value (eg comes from the egtb or actually any terminal position) then:
if win, don't average it in, but back up the absolute win value
if draw, back up the max(0.5, average so far)
if loss, back up the average so far

you can get extra clever and flag terminal nodes, then test status from the ply below. If there's one terminal win above then this ply is a terminal loss in all cases. if all terminal losses above, then this ply is a terminal win. and so on.
Maybe it's worth trying to backup TB-wins/-losses not only once, but several times?
To emphasize their impact. Needs testing, of course.

OTOH, from a theoretical point-of-view, I see no reason to treat TB nodes any different from terminal nodes.
Jörg Oster

Andrew
Posts: 99
Joined: Wed Mar 08, 2006 11:51 pm
Location: Australia

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Andrew » Wed Oct 03, 2018 10:26 am

Will these fixes be in the main LC0 release soon?

Andrew

Post Reply