Leela and the curse of not finishing a won match cleanly.

[OliverBr]

This game is another example, how mightily Leela plays the openings and mid-game and then... The first 30 moves are about a strategic dream for every chess player.
But then... instead of finished black with 31. Qc3+ she goes otherwise and trolls 180+ moves until checkmate.
So I decided to take action and let Stockfish 11 itself play it from Move 31:

[pgn][Event "Edited game"]
[Site "Olivers-MacBook-2.local"]
[Date "2020.06.08"]
[Round "-"]
[White "Lc0 0.25.1"]
[Black "Stockfish 11"]
[Result "1-0"]
[Annotator "1. +0.13 1... -0.09"]

1. e4 {+0.13/15} c5 {-0.09/24 9} 2. Nf3 {+0.16/16 4} e6 {+0.09/23 0.2} 3.
d4 {+0.19/16 4} cxd4 {+0.38/20 1.9} 4. Nxd4 {+0.20/18 0.7} Nf6 {+0.23/23 6}
5. Nc3 {+0.20/23 1.3} Nc6 {-0.03/24 6} 6. Nxc6 {+0.19/28 8} bxc6
{+0.30/21 2.1} 7. e5 {+0.20/28 0.3} Nd5 {+0.22/20 2.3} 8. Ne4
{+0.21/26 0.3} Qc7 {+0.00/22 6} 9. f4 {+0.21/28 0.3} Qb6 {+0.00/22 2.9} 10.
c4 {+0.19/25 0.4} Ne3 {-0.03/22 5} 11. Qd3 {+0.35/14 12} Bb4+ {-0.35/24 6}
12. Bd2 {+0.26/22 6} O-O {-0.24/23 0.3} 13. Rc1 {+0.48/14 8} Nxf1
{-0.38/28 10} 14. Rxf1 {+0.76/11 6} Be7 {-0.49/24 1.2} 15. Bc3 {+0.81/12 4}
Rd8 {-0.50/23 3} 16. Rd1 {+1.42/10 6} a5 {-1.15/29 19} 17. Rf3
{+1.54/13 0.5} Bb7 {-1.23/24 3} 18. Rg3 {+1.59/12 5} g6 {-1.24/24 13} 19.
Kf1 {+1.73/17 1.5} c5 {-1.25/24 8} 20. Nd6 {+2.31/15 6} Bxd6 {-1.70/25 1.2}
21. exd6 {+2.85/11 6} Qc6 {-2.25/26 0.2} 22. Re1 {+2.86/12 4} Ra6
{-1.63/22 2.1} 23. Be5 {+4.49/10 6} Rf8 {-1.82/25 3} 24. Ree3
{+5.51/10 2.5} f5 {-3.81/27 20} 25. Re2 {+6.10/11 9} Rf7 {-8.05/25 15} 26.
h4 {+15.00/7 6} Qc8 {-8.74/26 0.2} 27. h5 {+28.04/6 6} Rg7 {-9.40/29 0.2}
28. Bxg7 {+35.95/6 6} Kxg7 {-8.87/27 0.2} 29. hxg6 {+22.27/7 6} hxg6
{-10.02/30 8} 30. Rh3 {+18.26/8 4} Qg8 {-10.63/25 2.2} 31. Qc3+ {+10.20/31}
Kf7 {-10.81/30} 32. Rh8 {+10.38/33 0.1} Qg7 {-10.63/30 10} 33. Qh3
{+10.53/29 3} Rxd6 {-11.02/32 22} 34. Rh7 {+10.53/30 0.2} Rd1+
{-11.38/30 18} 35. Kf2 {+10.95/26 13} Rc1 {-12.02/34 36} 36. Rxg7+
{+12.08/30 0.2} Kxg7 {-12.03/30 10} 37. b3 {+12.24/28 10} Be4
{-12.30/29 32} 38. Qe3 {+13.96/27 25} Rh1 {-13.60/27 0.2} 39. Qxc5
{+14.47/28 33} Kf7 {-14.84/27 0.2} 40. Qxa5 {+15.81/28 40} Bc6
{-16.16/27 0.2} 41. b4 {+16.19/23 6} Rh7 {-18.62/26 56} 42. Qd8
{+24.70/24 8} Kg7 {-1000.15/28 24} 43. b5 {+1000.10/42 5} Be4
{-1000.09/43 3} 44. Qxd7+ {+1000.09/50 1.3} Kh6 {-1000.08/54 1.7} 45. Qd8
{+1000.08/57 3} Kg7 {-1000.07/57 0.7} 46. Rd2 {+1000.07/56 5} Bd5
{-1000.06/65 0.2} 47. cxd5 {+1000.06/63 4} Rh4 {-1000.05/98 0.2} 48. Qxh4
{+1000.05/104 4} e5 {-1000.04/245 0.2} 49. Rc2 {+1000.04/245 2.7} g5
{-1000.03/245 0.2} 50. Rc7+ {+1000.03/245 0.2} Kg8 {-1000.02/245 0.2} 51.
Qh6 {+1000.02/3 12} e4 {-1000.01/245} 52. Qg7# {+1000.01/2 17}
{Xboard adjudication: Checkmate} 1-0[/pgn]

No, surprise, Black is checkmate in move 52. Forced.

[Dann Corbit]

There are currently some solutions which play LC0 to start the game and automatically finish with Stockfish at some pre-arranged breaking point. Kai was very interested in that and did some experiments.

I think a better solution is an engine that fully utilizes both AB and NN. An interesting irony is that you can run them both at full throttle on a machine, because SF uses no GPU and LC0 only uses 3 threads with 2 x RTX 2080 Super GPUs. So on a machine with 64 threads, I often have SF running with 60 threads and LC0 running with 3 threads and both GPUs. This gives me one thread left for email, web browsing, etc.

The real solution will come with the next generation of infinity fabric. According to my understanding, it will give the GPU and CPU transparent access to the same memory.

Now, you can really cook with gas.

[Ovyron]

Solution to what? I only see two scenarios:

1. Winning games earlier doesn't matter. If it did, instead of 1 point awarded for a win, we'd have 1 for winning in 100 moves, 1.1 for winning on 90 moves, 1.2 for winning in 80 moves and so on. On this chess variant Leela would be learning to mate earlier to score more points.

2. The user doesn't want to waste resources with games that long when they could be playing more games.

For the 2., then:

a) So just adjudicate wins at certain point for Leela games - if you know it's going to win then just call it a win, and you don't have to waste resources if the game has been decided.

b) If you can't adjudicate the games because you're playing someone online, then who is wasting your resources is that person that refuses to resign and wastes their own resources and your resources instead of surrendering. But solving it comes from person to person, if the game is resigned then we don't need to worry about how long the game would have taken.

For a), then:

i - If you don't like adjudication because you're not certain about the game's outcome and you have to have to let it till the end and suffer the long wins, then the games where Leela doesn't win the won position ARE the actual problem, now how long she takes to win.

ii - If shorter wins are "cleaner" than longer ones, then not having adjudication is "dirty", but who's to blame about that? (hint: it's a 3 letter word that starts with y and ends with u with o in the middle.)

[Robert Pope]

What about
3. By playing an extra 100 pointless moves, it will occasionally wander into positions where an AB searcher can find a drawing continuation, turning a won game into a draw.

[carldaman]

Could there be an algorithmic solution to the unnecessarily drawn-out wins (or wins that become draws)?

For example, whenever Leela thinks it's winning big, why not temporarily 'pretend' the 50-move rule is really a 25-move rule, forcing the engine to look for faster wins, and if none are seen, then gradually and recursively lengthen the cutoff till eventually you default back to the 50-move rule, barring any faster wins?

Would this create havoc elsewhere? Unwanted or unexpected consequences?

Houdini had a feature like this, but best used for analysis, since Houdini 2 had frequent trouble making fast progress when it had the advantage. If you tried it in games, it could backfire and Houdini could walk into sudden losses where it was assuming it was safe because of the modified rule. But here, we're assuming Leela is winning big in the endgame.

[Dann Corbit]

The problem is caused by LC0 not searching deeply enough.
Searching less deeply won't fix that.

[MikeGL]

I agree, adjudication is the best route to stop the game earlier avoid reaching checkmate and avoid wasting cpu/gpu, and electric resources..
On the above specific case, stop Lc0 at move 26 after engine eval of both engines is over the roof.
On human eye, resignation or end of game is between move 36 to 39 where black is obviously hopeless.

...
...

2. The user doesn't want to waste resources with games that long when they could be playing more games.

For the 2., then:

a) So just adjudicate wins at certain point for Leela games - if you know it's going to win then just call it a win, and you don't have to waste resources if the game has been decided.
...
...
...

[Werewolf]

Slightly off-topic, but it is related.

Could this be used for Lc0

https://www.nvidia.com/en-gb/data-center/dgx-a100/

How many cards does Lc0 scale up to?

[Dann Corbit]

I don't think we know that max cards is for LC0.
I heard 2. Then I saw three running well.
Then I heard 3, and we see 4 in TCEC, with them running well.
I suspect it is a function of the OS, the cards, and LC0's cooperation algorithms.
Maybe someday we will have a better idea.
I wish it was not such a pain to set up.

[IanO]

Another possibility is to use stricter endgame counts which subsume the 50-move rule when training.

As an example, consider the Thai chess variant, Makruk. It has a host of different counts, which depend on the amount of material left.* Does anyone know if there is a project to apply the LC0 framework to Makruk?

* These counts only come into effect when the last pawn is traded off or promoted, so a NN might instead learn to preserve the last pawns as a general endgame technique.