Bughunting Games From Lichess

[jorose]

Recently Lichess added alpha support for engine accounts on their site. I created an account for Winter and let it play some games. Twice while kibitzing I was able to observe very strange moves that must be due to bugs, neither of which I have been able to reproduce and neither of which Winter approves at any depth. The moves I am referring to are 33. Ng6+ in https://lichess.org/yBIBt9QqU9Ky and 8. exf6 in https://lichess.org/jMhd6Wtm2hH2.

I have for a while suspected Winter must have an array of bugs but have not managed to find any in a while. In a way I am glad this occured as it seems to confirm my suspicions, unless it is possible that such moves are due to the API being buggy? My thinking is that the lack of reproduceability and the fact Winter does not approve of these moves by wide margins normally points to the issue being in the transposition table. Does this sound like a reasonable assumption or am I reading too much into this?

The Lichess bot API does not seem to make its own logs, would you recommend adding some form of logging to a UCI engine to try to hunt bugs of this form?

[Nordlandia]

At first i thought the title was Bughouse Games From Lichess

[jorose]

Here is yet another one.

https://lichess.org/EQegEeKR/white#68

[gbtami]

Recently Lichess added alpha support for engine accounts on their site. I created an account for Winter and let it play some games. Twice while kibitzing I was able to observe very strange moves that must be due to bugs, neither of which I have been able to reproduce and neither of which Winter approves at any depth. The moves I am referring to are 33. Ng6+ in https://lichess.org/yBIBt9QqU9Ky and 8. exf6 in https://lichess.org/jMhd6Wtm2hH2.

I have for a while suspected Winter must have an array of bugs but have not managed to find any in a while. In a way I am glad this occured as it seems to confirm my suspicions, unless it is possible that such moves are due to the API being buggy? My thinking is that the lack of reproduceability and the fact Winter does not approve of these moves by wide margins normally points to the issue being in the transposition table. Does this sound like a reasonable assumption or am I reading too much into this?

The Lichess bot API does not seem to make its own logs, would you recommend adding some form of logging to a UCI engine to try to hunt bugs of this form?

1. First change print_engine_stats to true (from false) at the bottom of your config.yml
2. In main.py change logging level from INFO to DEBUG in line
logger = logging.basicConfig(level=logging.INFO)

[jorose]

Thank you, hopefully this will make it easy to hunt for bugs if I find such a strange move again.

[jorose]

I didn't have to wait long for it to occur again... I wonder how often this occurs without me noticing it, since I am likely missing most occurrences unless they result in an extremely obvious blunder. My suspicion is that it is in the root of my search.

It seems that in advance Winter wanted to make the right move and promote a pawn to a queen with a completely winning position. Once it actually got there it didnt look at the promotion to a queen at any depth and preferred the promotion to a rook, with a bad or even losing position. I feel like I should be able to pinpoint the bug and I am glad it looks like it has nothing to do with the Lichess bot API as the commands look correct. Hopefully fixing it will result in a big elo gain

Code: Select all

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): lichess.org
DEBUG:urllib3.connectionpool:https://lichess.org:443 "POST /bot/game/UjmPmS5h/move/f4f5 HTTP/1.1" 200 None
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> << position fen 1r6/1P3kp1/P7/7p/5P1p/8/6PK/8 w - - 1 48 moves f4f5 f7f6
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> << go wtime 37740 btime 45640 winc 1000 binc 1000
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 1 seldepth 1 time 0 nodes 7 score cp 466 pv a6a7
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 2 seldepth 3 time 0 nodes 17 score cp 466 pv a6a7 b8b7
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 3 seldepth 3 time 1 nodes 28 score cp 466 pv a6a7 b8b7 a7a8q
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 4 seldepth 4 time 1 nodes 41 score cp 466 pv a6a7 b8b7 a7a8q b7b6
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 5 seldepth 6 time 1 nodes 60 score cp 466 pv a6a7 b8b7 a7a8q b7b6 a8d8
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 6 seldepth 7 time 1 nodes 109 score cp 466 pv a6a7 b8b7 a7a8q b7b6 a8d8 f6f5
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 7 seldepth 9 time 2 nodes 211 score cp 466 pv a6a7 b8b7 a7a8q b7b6 a8d8 f6f5 d8h4
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 8 seldepth 9 time 3 nodes 452 score cp 466 pv a6a7 b8b7 a7a8q b7b6 a8d8 f6f5 d8h4 b6h6
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 9 seldepth 11 time 6 nodes 907 score cp 466 pv a6a7 b8b7 a7a8q b7b6 a8d8 f6f5 d8h4 b6h6 h4g3
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 10 seldepth 13 time 11 nodes 1812 score cp 466 pv a6a7 b8b7 a7a8q b7b6 a8d8 f6f5 d8h4 b6h6 h4g3 g7g5
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 11 seldepth 13 time 46 nodes 3578 score cp 466 pv a6a7 b8b7 a7a8q b7b6 a8d8 f6f5 d8h4 b6h6 h4g3 g7g5 g3c3
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 12 seldepth 16 time 65 nodes 6912 score cp 466 pv a6a7 b8b7 a7a8q b7b6 a8d8 f6f5 d8h4 b6h6 h4g3 g7g5 g3c3 h6d6
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 13 seldepth 20 time 198 nodes 138640 score cp 494 pv a6a7 b8b7 a7a8q b7d7 a8e8 d7d4 e8h5 f6e5 h5g5 d4f4 g5g7 e5f5 g7c7
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 14 seldepth 21 time 359 nodes 316218 score cp 481 pv a6a7 b8b7 a7a8q b7d7 a8e4 d7d6 e4f4 d6d5 f4h4 g7g5 f5g6 f6g6 h4e4 d5f5
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 15 seldepth 23 time 672 nodes 767347 score cp 510 pv a6a7 b8b7 a7a8q b7d7 a8e8 d7e7 e8g6 f6e5 g6h5 e7c7 h5g5 e5e4 f5f6 g7f6 g5f6
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 16 seldepth 23 time 980 nodes 1220090 score cp 510 pv a6a7 b8b7 a7a8q b7d7 a8e8 d7e7 e8g6 f6e5 g6h5 e7c7 h5g5 e5e4 f5f6 g7f6 g5f6 c7h7
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 17 seldepth 26 time 1711 nodes 2315344 score cp 520 pv a6a7 b8b7 a7a8q b7d7 a8e8 d7e7 e8g6 f6e5 g6h5 e7c7 h5h4 e5f5 h4f2 f5e6 f2b6 e6d7 b6g6
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> bestmove a6a7
DEBUG&#58;urllib3.connectionpool&#58;Starting new HTTPS connection &#40;1&#41;&#58; lichess.org
DEBUG&#58;urllib3.connectionpool&#58;https&#58;//lichess.org&#58;443 "POST /bot/game/UjmPmS5h/move/a6a7 HTTP/1.1" 200 None
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> << position fen 1r6/PP4p1/5k2/5P1p/7p/8/6PK/8 b - - 0 49 moves b8b7
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> << go wtime 37400 btime 45650 winc 1000 binc 1000
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 1 seldepth 4 time 0 nodes 27 score cp -165 pv a7a8r
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 2 seldepth 4 time 0 nodes 87 score cp -165 pv a7a8r f6f5
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 3 seldepth 6 time 1 nodes 244 score cp -143 pv a7a8r f6f5 a8a5
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 4 seldepth 6 time 3 nodes 618 score cp -143 pv a7a8r f6f5 a8a5 f5g6
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 5 seldepth 8 time 13 nodes 2288 score cp -132 pv a7a8r f6f5 a8a5 f5g6 a5e5
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 6 seldepth 10 time 31 nodes 5284 score cp -140 pv a7a8r f6f5 a8a5 f5g6 a5e5 b7b6
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 7 seldepth 12 time 73 nodes 12449 score cp -139 pv a7a8r f6f5 a8a5 f5g6 a5e5 b7f7 e5e4
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 8 seldepth 13 time 127 nodes 24181 score cp -146 pv a7a8r f6f5 a8a5 f5g6 a5a4 g6g5 a4a5 g5g6
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 9 seldepth 14 time 156 nodes 51050 score cp -135 pv a7a8r f6f5 a8a5 f5g6 a5a4 g6g5 a4a5 g5h6 a5e5
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 10 seldepth 14 time 184 nodes 93662 score cp -135 pv a7a8r f6f5 a8a5 f5g6 a5a4 g6g5 a4a5 g5h6 a5a4 g7g5
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 11 seldepth 16 time 254 nodes 181497 score cp -135 pv a7a8r f6f5 a8a5 f5g6 a5a4 g6g5 a4a5 g5h6 a5a4 g7g5 a4d4
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 12 seldepth 17 time 364 nodes 350742 score cp -138 pv a7a8r f6f5 a8a5 f5g6 a5a4 g6g5 a4a5 g5h6 a5a4 g7g5 a4e4 h6g6
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 13 seldepth 19 time 564 nodes 657242 score cp -138 pv a7a8r f6f5 a8a5 f5g6 a5a4 g6g5 a4a5 g5h6 a5a4 g7g5 a4e4 h6g6 e4e5
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 14 seldepth 21 time 1015 nodes 1319964 score cp -148 pv a7a8r f6f5 a8a5 f5g6 a5a4 g6g5 a4a5 g5h6 a5a4 g7g5 a4e4 b7f7 e4e5 f7f2
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> info  depth 15 seldepth 21 time 1607 nodes 2253151 score cp -148 pv a7a8r f6f5 a8a5 f5g6 a5a4 g6g5 a4a5 g5h6 a5a4 g7g5 a4e4 b7f7 e4e5 f7f2 e5e4
DEBUG&#58;chess.engine&#58;<PopenProcess at 0x7f7849d9edd8 &#40;pid=22959&#41;> >> bestmove a7a8r

[gbtami]

It's nice to see how lichess, lichess-bot and python-chess can help engine authors to find bugs