Lc0 v18 rc2 for 2080 Syzygy implementation

[Damir]

This happens when playing on Playchess server... I don't know why, but it happens… Maybe Leela has bug located in its time code when playing 16+0, or much longer games like 60/15 , which cause it to suddenly resign the game before it is even finished?

[jkiliani]

I tested yesterday on 100 long 6-men draws (not sure if they are exactly hard, but many of them are), no losses. Avoiding TB losses against non-TB engine needs comparison with another Syzygy-enabled engine against that non-TB engine. Might compare Lc0_Syzygy against SF8_no_TB compared to SF8_Syzygy against SF8_no_TB on 6-men hard wins. Will depend on time control used too.

Out of 50 6-men TB losses, Lc0_Syzygy (the fixed one) saves 13 against SF8_no_TB, SF8_Syzygy saves 15 against SF8_no_TB. So, behaves similarly to SF8, which has a good TB implementation (and good endgame eval, which Lc0 doesn't). Might check how many saves SF dev.

SF_dev_Syzygy saves 23 out of 50 TB losses. Might be due to different Syzygy implementation, stronger play, etc., I don't know.

Nice experiment, thanks! If Leela can play TB wins and draws perfectly and avoid some TB losses against no-TB opponents, the implementation can probably be considered a success. All engines strong enough for a reasonable chance to reach a TB win against Leela have TB themselves, so the loss avoidance likely isn't that important.

In a few months when we have strong Leela networks based on TB rescoring, it will be interesting how those will score in these benchmarks without using TB themselves...

[Laskos]

I tested yesterday on 100 long 6-men draws (not sure if they are exactly hard, but many of them are), no losses. Avoiding TB losses against non-TB engine needs comparison with another Syzygy-enabled engine against that non-TB engine. Might compare Lc0_Syzygy against SF8_no_TB compared to SF8_Syzygy against SF8_no_TB on 6-men hard wins. Will depend on time control used too.

Out of 50 6-men TB losses, Lc0_Syzygy (the fixed one) saves 13 against SF8_no_TB, SF8_Syzygy saves 15 against SF8_no_TB. So, behaves similarly to SF8, which has a good TB implementation (and good endgame eval, which Lc0 doesn't). Might check how many saves SF dev.

SF_dev_Syzygy saves 23 out of 50 TB losses. Might be due to different Syzygy implementation, stronger play, etc., I don't know.

Nice experiment, thanks! If Leela can play TB wins and draws perfectly and avoid some TB losses against no-TB opponents, the implementation can probably be considered a success. All engines strong enough for a reasonable chance to reach a TB win against Leela have TB themselves, so the loss avoidance likely isn't that important.

In a few months when we have strong Leela networks based on TB rescoring, it will be interesting how those will score in these benchmarks without using TB themselves...

I checked for another set of 400 won 6-men positions against SF8_Syzygy, not a single miss.
TBs are only up to 7 men, Leela often misevaluates heavily earlier endgames with many more pieces, I am bit skeptical that based on TB rescoring, Leela will become comparable to top engines in endgames, with their hand-crafted knowledge, which is often general and pretty abstract.

[Milos]

I checked for another set of 400 won 6-men positions against SF8_Syzygy, not a single miss.
TBs are only up to 7 men, Leela often misevaluates heavily earlier endgames with many more pieces, I am bit skeptical that based on TB rescoring, Leela will become comparable to top engines in endgames, with their hand-crafted knowledge, which is often general and pretty abstract.

You scepticism is totally in place. 6-men at root solves almost nothing for Lc0.
Problem is adding 6-men properly to leaves is also not working due to MCTS using averaging as an operator. There is no easy solution. Maybe some hybrid approach that would switch operator to minmax once you start reaching TB positions at leaves, but that is not easy to implement and even harder to tune.

[chrisw]

I checked for another set of 400 won 6-men positions against SF8_Syzygy, not a single miss.
TBs are only up to 7 men, Leela often misevaluates heavily earlier endgames with many more pieces, I am bit skeptical that based on TB rescoring, Leela will become comparable to top engines in endgames, with their hand-crafted knowledge, which is often general and pretty abstract.

You scepticism is totally in place. 6-men at root solves almost nothing for Lc0.
Problem is adding 6-men properly to leaves is also not working due to MCTS using averaging as an operator. There is no easy solution. Maybe some hybrid approach that would switch operator to minmax once you start reaching TB positions at leaves, but that is not easy to implement and even harder to tune.

it's a relatively simple modification to the backed up value. if you ask for a value at a leaf, and the value is an absolute value (eg comes from the egtb or actually any terminal position) then:
if win, don't average it in, but back up the absolute win value
if draw, back up the max(0.5, average so far)
if loss, back up the average so far

you can get extra clever and flag terminal nodes, then test status from the ply below. If there's one terminal win above then this ply is a terminal loss in all cases. if all terminal losses above, then this ply is a terminal win. and so on.

[Milos]

I checked for another set of 400 won 6-men positions against SF8_Syzygy, not a single miss.
TBs are only up to 7 men, Leela often misevaluates heavily earlier endgames with many more pieces, I am bit skeptical that based on TB rescoring, Leela will become comparable to top engines in endgames, with their hand-crafted knowledge, which is often general and pretty abstract.

You scepticism is totally in place. 6-men at root solves almost nothing for Lc0.
Problem is adding 6-men properly to leaves is also not working due to MCTS using averaging as an operator. There is no easy solution. Maybe some hybrid approach that would switch operator to minmax once you start reaching TB positions at leaves, but that is not easy to implement and even harder to tune.

it's a relatively simple modification to the backed up value. if you ask for a value at a leaf, and the value is an absolute value (eg comes from the egtb or actually any terminal position) then:
if win, don't average it in, but back up the absolute win value
if draw, back up the max(0.5, average so far)
if loss, back up the average so far

you can get extra clever and flag terminal nodes, then test status from the ply below. If there's one terminal win above then this ply is a terminal loss in all cases. if all terminal losses above, then this ply is a terminal win. and so on.

Again, not so easy. You backup lets say a win, move one ply back to the root and do what, since that becomes the worst move for your opponent you just average it up and loose any kind of information. It doesn't propagate further up the tree.
The only thing that is actually easy to implement and might be able to work is to give higher weight to terminal positions when averaging. But that would require a lot of tuning.

[chrisw]

I checked for another set of 400 won 6-men positions against SF8_Syzygy, not a single miss.
TBs are only up to 7 men, Leela often misevaluates heavily earlier endgames with many more pieces, I am bit skeptical that based on TB rescoring, Leela will become comparable to top engines in endgames, with their hand-crafted knowledge, which is often general and pretty abstract.

You scepticism is totally in place. 6-men at root solves almost nothing for Lc0.
Problem is adding 6-men properly to leaves is also not working due to MCTS using averaging as an operator. There is no easy solution. Maybe some hybrid approach that would switch operator to minmax once you start reaching TB positions at leaves, but that is not easy to implement and even harder to tune.

it's a relatively simple modification to the backed up value. if you ask for a value at a leaf, and the value is an absolute value (eg comes from the egtb or actually any terminal position) then:
if win, don't average it in, but back up the absolute win value
if draw, back up the max(0.5, average so far)
if loss, back up the average so far

you can get extra clever and flag terminal nodes, then test status from the ply below. If there's one terminal win above then this ply is a terminal loss in all cases. if all terminal losses above, then this ply is a terminal win. and so on.

Again, not so easy. You backup lets say a win, move one ply back to the root and do what, since that becomes the worst move for your opponent you just average it up and loose any kind of information. It doesn't propagate further up the tree.

it is a bit of a hassle to get it to work, you are kind of minimaxing, but only in terminal situations, and you have to treat win nodes differently to loss nodes.
a win node backs up to a "terminal" loss the ply below. end of story. (and an immediate algorithmic gain over averaging)
a loss node only backs up to a "terminal" win one ply below if all siblings are lost. so you need to keep some sort of flagging count.
Think about it, you can make it work.

[Laskos]

I tested yesterday on 100 long 6-men draws (not sure if they are exactly hard, but many of them are), no losses. Avoiding TB losses against non-TB engine needs comparison with another Syzygy-enabled engine against that non-TB engine. Might compare Lc0_Syzygy against SF8_no_TB compared to SF8_Syzygy against SF8_no_TB on 6-men hard wins. Will depend on time control used too.

Out of 50 6-men TB losses, Lc0_Syzygy (the fixed one) saves 13 against SF8_no_TB, SF8_Syzygy saves 15 against SF8_no_TB. So, behaves similarly to SF8, which has a good TB implementation (and good endgame eval, which Lc0 doesn't). Might check how many saves SF dev.

SF_dev_Syzygy saves 23 out of 50 TB losses. Might be due to different Syzygy implementation, stronger play, etc., I don't know.

Nice experiment, thanks! If Leela can play TB wins and draws perfectly and avoid some TB losses against no-TB opponents, the implementation can probably be considered a success. All engines strong enough for a reasonable chance to reach a TB win against Leela have TB themselves, so the loss avoidance likely isn't that important.

Yes, the implementation is fine and does help a lot, especially the DTZ part. Just compare to what Lc0 does without Syzygy on 6-men TB positions at root:

Out of 50 6-men TB wins, Lc0_no_TBs against SF_dev_Syzygy converts 6, misses as draws 38, and misses even as losses 6. So, from 6-men won positions, Lc0_no_TB scores 50% against SF_dev_Syzygy from 6-men wins!

So, the Syzygy implementation is fine and helps a lot in this case, as Lc0 plays really badly in these late endgames.

[Joerg Oster]

I checked for another set of 400 won 6-men positions against SF8_Syzygy, not a single miss.
TBs are only up to 7 men, Leela often misevaluates heavily earlier endgames with many more pieces, I am bit skeptical that based on TB rescoring, Leela will become comparable to top engines in endgames, with their hand-crafted knowledge, which is often general and pretty abstract.

You scepticism is totally in place. 6-men at root solves almost nothing for Lc0.
Problem is adding 6-men properly to leaves is also not working due to MCTS using averaging as an operator. There is no easy solution. Maybe some hybrid approach that would switch operator to minmax once you start reaching TB positions at leaves, but that is not easy to implement and even harder to tune.

it's a relatively simple modification to the backed up value. if you ask for a value at a leaf, and the value is an absolute value (eg comes from the egtb or actually any terminal position) then:
if win, don't average it in, but back up the absolute win value
if draw, back up the max(0.5, average so far)
if loss, back up the average so far

you can get extra clever and flag terminal nodes, then test status from the ply below. If there's one terminal win above then this ply is a terminal loss in all cases. if all terminal losses above, then this ply is a terminal win. and so on.

Maybe it's worth trying to backup TB-wins/-losses not only once, but several times?
To emphasize their impact. Needs testing, of course.

OTOH, from a theoretical point-of-view, I see no reason to treat TB nodes any different from terminal nodes.

[Andrew]

Will these fixes be in the main LC0 release soon?

Andrew