Interesting game from CCC 5: Fire - Lc0

[jdart]

Here is an example where Lc0 seems to have had the better evaluation.

Fire traded Rooks and went into a Pawn + Bishop endgame, which was maybe a bad choice. Then it kept pushing pawns. I think in particular c5 was bad here:

[d]8/5p2/1p2pk2/pP3p1P/2P2b2/1P5K/7P/3B4 w - - 0 90

92. b7 was even worse:

[d]8/5p2/1P2p3/p1p2pkP/5b2/1P5K/7P/3B4 w - - 0 92

White gets an advanced passer, but it can be blocked, and it is not worth giving up a pawn for.

[pgn][Event "CCC 5: Escalation (10|5)"] [Site "?"] [Date "2019.02.02"] [Round "1"] [White "Fire"] [Black "Lc0-dev"] [Result "0-1"] [ECO "C07"] [PlyCount "348"] [EventDate "2019.??.??"] 1. e4 e6 2. d4 d5 3. Nd2 c5 4. exd5 Qxd5 5. Ngf3 cxd4 6. Bc4 Qd6 7. O-O Nf6 8. Nb3 Nc6 9. Nbxd4 Nxd4 10. Nxd4 Be7 11. Bb3 a5 12. c3 O-O 13. a4 Qb6 14. Qe2 Rd8 15. Nb5 Bd7 16. Bf4 Bc5 17. Rad1 Bxb5 18. axb5 Bd6 19. Bg5 Qc7 20. g3 h6 21. Bxf6 gxf6 22. Rd4 f5 23. Rc4 Qb6 24. Re1 Bf8 25. Rf4 Qd6 26. Rd4 Qe7 27. Kf1 Rac8 28. Red1 b6 29. R4d3 Rxd3 30. Rxd3 Rc5 31. Rd2 Qb7 32. f3 Qc7 33. Kg2 Re5 34. Qd3 Rc5 35. Kh3 Qc8 36. Qe2 Qe8 37. c4 Rc7 38. Qd3 Bb4 39. Rd1 Bc5 40. f4 Qe7 41. Bc2 Rc8 42. b3 Qf6 43. Qd7 Rb8 44. Qc7 Rf8 45. Qc6 Kg7 46. Qf3 Re8 47. Kg2 Bd4 48. Rd3 Bc5 49. Qd1 Rh8 50. Rd7 Re8 51. Qd3 Qb2 52. Qd2 Ra8 53. Qe2 Re8 54. Kh3 h5 55. Qg2 Rh8 56. Qe2 Qf6 57. Bd1 Rh6 58. Qe1 Qb2 59. Qe2 Qc3 60. Qd2 Qf6 61. Qe1 Qb2 62. Qe2 Qc3 63. Qd2 Qxd2 64. Rxd2 h4 65. gxh4 Rh8 66. Kg3 Rg8 67. Kh3 Kf6 68. Rd7 Rg1 69. Bh5 Rg7 70. Bf3 Rg1 71. Bh5 Rg7 72. Rd3 Ke7 73. Bf3 Rg8 74. h5 Bd6 75. Rd4 Rh8 76. Kh4 Ba3 77. Rd1 Bc5 78. Rd3 Ba3 79. Be2 Bd6 80. Rd4 Bc5 81. Rd1 Be3 82. Rf1 Bc5 83. Rf3 Bd6 84. Rf1 Rg8 85. Kh3 Kf6 86. Bd1 Rd8 87. Be2 Bc7 88. Rd1 Rxd1 89. Bxd1 Bxf4 90. c5 bxc5 91. b6 Kg5 92. b7 Bb8 93. Be2 e5 94. Bc4 f6 95. Bf7 f4 96. Kg2 e4 97. h4+ Kh6 98. Bg6 c4 99. bxc4 f5 100. Bxf5 e3 101. Kf3 a4 102. Bg6 a3 103. Bb1 Kxh5 104. Ba2 Kxh4 105. Bb3 Kg5 106. Ba2 Kf5 107. Bb1+ Ke5 108. Ba2 Kd4 109. Ke2 Kc3 110. c5 Kb2 111. Bd5 a2 112. Bxa2 Kxa2 113. c6 Kb3 114. Kf3 Kc4 115. Kg4 e2 116. Kf3 e1=Q 117. c7 Bxc7 118. Kg4 Qg1+ 119. Kf5 Qb1+ 120. Kf6 Qa2 121. b8=Q Bxb8 122. Kg7 Qa3 123. Kf7 Qa1 124. Kg6 Qb2 125. Kf7 Qc1 126. Ke6 Qe1+ 127. Kf7 Qb1 128. Kf6 Qa2 129. Kg7 f3 130. Kf6 f2 131. Kf5 Qa3 132. Kg4 Qa1 133. Kf5 Qd1 134. Kg5 Qb1 135. Kh6 Qa2 136. Kg5 Qa3 137. Kh6 Qa1 138. Kg5 Qd1 139. Kh6 Qb1 140. Kg5 Qa2 141. Kf5 f1=Q+ 142. Ke6 Qae2+ 143. Kd7 Qe7+ 144. Kxe7 Qc1 145. Ke6 Qe1+ 146. Kf5 Qb1+ 147. Kg5 Qa2 148. Kg6 Qa3 149. Kf6 Qa1+ 150. Kg5 Qb2 151. Kf5 Qc2+ 152. Kf6 Qd1 153. Kg5 Qb1 154. Kf6 Qa2 155. Kf7 Qa3 156. Kf6 Bd6 157. Ke6 Qf3 158. Kxd6 Qd1+ 159. Ke5 Qb1 160. Ke6 Qa2 161. Kf6 Qa3 162. Kf5 Qa1 163. Ke6 Qb2 164. Kf5 Qc1 165. Kg4 Qc2 166. Kg5 Qd1 167. Kf6 Qb1 168. Ke6 Kd4 169. Kd6 Qe1 170. Kc6 Qe7 171. Kb6 Kd5 172. Ka5 Kc5 173. Ka4 Qe3 174. Ka5 Qa3# 0-1 [/pgn]

[mar]

A nice win, except for the endgame. Giving up extra queen for free?
Leela plays endgames even worse than Skipper!
Can't see mate in 3 here? [d]1b6/8/4K3/8/2k5/8/q7/5q2 b - - 1 142
Torturing the audience with 30 more moves

[Nay Lin Tun]

The losing move was not c5.
It was Rd1?? Quite strange SF also stacked with Rd1??(after 500 millions nodes).
As a human player's positional point of view, I see only two weakness for white, "f4" and "b3" both could be defend-able by keeping Rook , with Rf3.
I played against SF whether SF can beat my fortress with Rf3, but can't (after about 500 million nodes).

I think these are the reasons AB engines are down-falling with very bad positional evaluation. Humans have very strong positional evaluation but cant compete x1,000,000 times faster AB engines. However, human like positional evaluation with x1000 faster Leela can compete AB engines now.

[yanquis1972]

T40s endgame eval in general already seems phenomenal. Does anyone know of a good endgame test suite? I’m curious how it stacks up to T30 minus TBs.

[M ANSARI]

A nice win, except for the endgame. Giving up extra queen for free?
Leela plays endgames even worse than Skipper!
Can't see mate in 3 here? [d]1b6/8/4K3/8/2k5/8/q7/5q2 b - - 1 142
Torturing the audience with 30 more moves

Yes, this is something that needs to be looked at. I think this is also why Lc0 can blunder badly in some positions. Not sure what it is ... is it search related or is it due to the way its pruning works? Or is it just something you have to live with with an NN engine?

[carldaman]

A nice win, except for the endgame. Giving up extra queen for free?
Leela plays endgames even worse than Skipper!
Can't see mate in 3 here? [d]1b6/8/4K3/8/2k5/8/q7/5q2 b - - 1 142
Torturing the audience with 30 more moves

Yes, this is something that needs to be looked at. I think this is also why Lc0 can blunder badly in some positions. Not sure what it is ... is it search related or is it due to the way its pruning works? Or is it just something you have to live with with an NN engine?

What's puzzling to me is that Leela's developers are quite aware of the sloppy endgame problem, but they can't, or won't, fix it - I forget which.

[M ANSARI]

Who are the developers of Leela? I would like to get involved and see if I can contribute to helping them.

[Ovyron]

Who are the developers of Leela? I would like to get involved and see if I can contribute to helping them.

I think the best way to get involved is to join the Leela Discord? https://discord.gg/pKujYxD

[jdart]

What's puzzling to me is that Leela's developers are quite aware of the sloppy endgame problem, but they can't, or won't, fix it - I forget which.

As I understand, they are very committed to a "pure NN" approach. The effective nps for this is very low. They are relying on the NN's pattern recognition and that is only as good as it is trained to be; it will steer the program wrong in some situations, which is of course also true for the eval of an A/B searcher. But the difference is, an A/B searcher never misses mate in 3. So adding a mate finder to the engine would of course avoid this issue (although there would still be other instances of blindness), but I don't think they want to do that.

--Jon