Did Crafty make an illegal move, or is cutechess dumb?

[jkominek]

I'm playing a game in cutechess_cli between Stockfish 15.1 and Crafty 18.0. (Crafty has a bigger version number, so it's gotta be stronger, right?)

Cutechess adjudicates the game as a win for Stockfish on the grounds that Crafty made an illegal move. The game's final (intended) move for black is Rexd2+ with discovered check, but what was sent to the UI was Rxd2+. With a second black rook on b2 the move notation would be ambiguous, except that the check status disambiguates the two potential rook captures.

Is Crafty at fault for not being pedantic? Or is cutechess at fault for not applying available constraints? I'm not posing this as a quiz. I don't know the answer.

Here is the game.

[pgn]
[Event "chess324_std.SF15.1_Crafty18.0.n2048_tc20+1"]
[Site "2x Pentium III @ 1.0 GHz"]
[Date "2023.02.26"]
[Round "1"]
[White "Stockfish 15.1 Hash 128 Threads 1 Nodes 2048"]
[Black "Crafty 18.0 Hash 128 Threads 1 2x Pentium III TC 20+1"]
[Result "1-0"]
[BlackTimeControl "20+1"]
[FEN "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1"]
[GameDuration "00:00:55"]
[GameEndTime "2023-02-26T19:37:28.567 EST"]
[GameStartTime "2023-02-26T19:36:33.279 EST"]
[PlyCount "93"]
[SetUp "1"]
[Termination "illegal move"]
[WhiteTimeControl "inf"]

1. e4 {+0.34/7 0.10s hf=0 tb=0 n=2050 nps=20297}
d5 {-0.17/7 1.6s hf=0 tb=0 n=110920 nps=71515}
2. exd5 {+0.68/9 0.085s hf=0 tb=0 n=2048 nps=24094}
Qxd5 {-0.25/8 1.2s hf=0 tb=0 n=356525 nps=300105}
3. Nc3 {+0.65/9 0.15s hf=0 tb=0 n=2050 nps=14236}
Qd8 {-0.24/7 1.0s hf=0 tb=0 n=383293 nps=365737}
4. d4 {+0.76/8 0.10s hf=0 tb=0 n=2050 nps=20500}
Nf6 {-0.23/7 1.0s hf=0 tb=0 n=253072 nps=252819}
5. Nf3 {+0.75/7 0.11s hf=0 tb=0 n=2051 nps=18816}
e6 {-0.32/7 1.0s hf=0 tb=0 n=301835 nps=292475}
6. Bc4 {+0.52/7 0.10s hf=0 tb=0 n=2049 nps=20696}
c6 {-0.32/7 1.0s hf=0 tb=0 n=357678 nps=352045}
7. Qe2 {+0.80/7 0.13s hf=0 tb=0 n=2048 nps=16000}
b5 {-0.40/8 1.2s hf=0 tb=0 n=276124 nps=232427}
8. Nxb5 {+1.68/9 0.11s hf=0 tb=0 n=2051 nps=19168}
cxb5 {+1.24/7 1.0s hf=0 tb=0 n=127699 nps=125072}
9. Bxb5+ {-1.85/9 0.083s hf=0 tb=0 n=2049 nps=24987}
Bd7 {+1.26/8 1.0s hf=0 tb=0 n=345776 nps=345085}
10. Bd3 {-1.75/8 0.089s hf=0 tb=0 n=2050 nps=23295}
Bd6 {+1.52/7 1.0s hf=0 tb=0 n=298992 nps=297504}
11. O-O {-1.70/8 0.090s hf=0 tb=0 n=2048 nps=23011}
Nc6 {+1.52/8 1.0s hf=0 tb=0 n=245040 nps=245040}
12. c4 {-1.78/8 0.084s hf=0 tb=0 n=2049 nps=24392}
O-O {+1.60/7 1.0s hf=0 tb=0 n=168274 nps=164169}
13. a3 {-1.61/9 0.086s hf=0 tb=0 n=2049 nps=24105}
a5 {+1.60/7 1.9s hf=0 tb=0 n=448804 nps=235716}
14. Bg5 {-1.71/9 0.097s hf=0 tb=0 n=2049 nps=21343}
h6 {+1.70/7 1.0s hf=0 tb=0 n=366870 nps=358621}
15. Bh4 {-1.39/8 0.093s hf=0 tb=0 n=2049 nps=22271}
a4 {+1.72/8 4.9s hf=0 tb=0 n=2018590 nps=413984}
16. Rad1 {-1.19/8 0.10s hf=0 tb=0 n=2048 nps=20480}
Be8 {+1.52/8 3.2s hf=0 tb=0 n=1298244 nps=407483}
17. Bb1 {-0.46/9 0.094s hf=0 tb=0 n=2048 nps=21787}
g5 {+2.10/7 0.50s hf=0 tb=0 n=171310 nps=344688}
18. Nxg5 {+0.13/10 0.093s hf=0 tb=0 n=2048 nps=22021}
hxg5 {+1.94/7 0.60s hf=0 tb=0 n=171308 nps=286468}
19. Bxg5 {-0.83/8 0.070s hf=0 tb=0 n=2049 nps=29271}
Nxd4 {+1.76/6 0.50s hf=0 tb=0 n=187857 nps=372732}
20. Qd3 {+3.30/9 0.10s hf=0 tb=0 n=2048 nps=20686}
Nf5 {+1.20/6 0.50s hf=0 tb=0 n=156111 nps=309744}
21. Qh3 {+1.16/9 0.087s hf=0 tb=0 n=2049 nps=24105}
Ra6 {+0.61/6 0.50s hf=0 tb=0 n=160292 nps=321226}
22. Bxf5 {+2.81/9 0.12s hf=0 tb=0 n=2048 nps=16650}
exf5 {+0.63/8 1.0s hf=0 tb=0 n=490859 nps=478886}
23. Qxf5 {-2.08/8 0.082s hf=0 tb=0 n=2048 nps=25600}
Qb8 {+0.92/7 1.0s hf=0 tb=0 n=389357 nps=377649}
24. Bxf6 {-0.42/9 0.078s hf=0 tb=0 n=2050 nps=26973}
Bxh2+ {+0.56/8 1.0s hf=0 tb=0 n=322359 nps=319167}
25. Kh1 {-1.37/10 0.074s hf=0 tb=0 n=2048 nps=28054}
Qf4 {+0.76/9 1.00s hf=0 tb=0 n=223514 nps=224411}
26. Qg5+ {-2.28/10 0.075s hf=0 tb=0 n=2048 nps=27675}
Qxg5 {+0.88/10 1.0s hf=0 tb=0 n=393493 nps=385022}
27. Bxg5 {-2.28/9 0.097s hf=0 tb=0 n=2048 nps=21557}
Bc7 {+0.93/9 1.0s hf=0 tb=0 n=298416 nps=298416}
28. Rfe1 {-2.13/8 0.093s hf=0 tb=0 n=2048 nps=22505}
Kg7 {1.00s hf=0 tb=0 n=0 nps=0}
29. Bd2 {-1.67/8 0.088s hf=0 tb=0 n=2049 nps=23551}
Rc6 {+1.11/8 1.0s hf=0 tb=0 n=514206 nps=507607}
30. Bc3+ {-1.44/8 0.093s hf=0 tb=0 n=2048 nps=22505}
Kg6 {+1.26/9 1.0s hf=0 tb=0 n=463944 nps=462095}
31. Rd4 {-1.77/9 0.083s hf=0 tb=0 n=2052 nps=25024}
f5 {+1.25/8 1.0s hf=0 tb=0 n=247264 nps=246033}
32. Re7 {-0.56/9 0.090s hf=0 tb=0 n=2048 nps=23272}
Bf7 {+1.37/8 1.0s hf=0 tb=0 n=500696 nps=499198}
33. Rdd7 {-0.80/7 0.083s hf=0 tb=0 n=2049 nps=24987}
Bb8 {+1.93/8 1.9s hf=0 tb=0 n=1043467 nps=535385}
34. g3 {-1.06/7 0.081s hf=0 tb=0 n=2048 nps=25600}
Rxc4 {+2.21/7 0.91s hf=0 tb=0 n=463638 nps=508375}
35. f4 {-1.34/8 0.069s hf=0 tb=0 n=2048 nps=29681}
Rc6 {+2.40/7 0.50s hf=0 tb=0 n=250380 nps=503782}
36. Bb4 {-1.21/8 0.078s hf=0 tb=0 n=2048 nps=26597}
Rc1+ {+2.92/7 0.60s hf=0 tb=0 n=328005 nps=545765}
37. Kg2 {-0.70/10 0.091s hf=0 tb=0 n=2048 nps=22755}
Rc2+ {1.0s hf=0 tb=0 n=0 nps=0}
38. Kg1 {-1.49/8 0.088s hf=0 tb=0 n=2051 nps=23306}
Rxb2 {+3.25/8 1.0s hf=0 tb=0 n=334356 nps=333355}
39. Rb7 {-1.60/8 0.090s hf=0 tb=0 n=2052 nps=23318}
Rc8 {+4.03/8 1.4s hf=0 tb=0 n=798021 nps=565972}
40. Rbd7 {-5.52/8 0.079s hf=0 tb=0 n=2048 nps=26256}
Rcc2 {+7.26/7 0.50s hf=0 tb=0 n=295236 nps=590472}
41. g4 {-3.95/8 0.093s hf=0 tb=0 n=2049 nps=22032}
Rg2+ {+13.70/7 1.0s hf=0 tb=0 n=255363 nps=253336}
42. Kf1 {-6.03/8 0.070s hf=0 tb=0 n=2048 nps=29257}
Bc4+ {+15.19/8 1.0s hf=0 tb=0 n=269643 nps=264355}
43. Re2 {-6.06/8 0.050s hf=1 tb=0 n=2049 nps=41816}
Rgxe2 {+327.52/7 1.0s hf=0 tb=0 n=400766 nps=399966}
44. gxf5+ {-7.29/8 0.025s hf=0 tb=0 n=2057 nps=85708}
Kh5 {+327.56/8 1.0s hf=0 tb=0 n=599791 nps=590927}
45. Rh7+ {-7.08/8 0.036s hf=0 tb=0 n=2048 nps=58514}
Kg4 {+327.58/4 0.004s hf=0 tb=0 n=5 nps=1250}
46. Rg7+ {-M14/7 0.025s hf=0 tb=0 n=2048 nps=85333}
Kxf4 {+327.60/7 0.73s hf=0 tb=0 n=494502 nps=678329}
47. Bd2+ {-M6/11 0.035s hf=0 tb=0 n=2049 nps=62090, Black makes an illegal move: Rxd2+}
1-0
[/pgn]

[AndrewGrant]

Could this be a very interesting SAN case? So two rooks were able to make that capture. Rd2+ is only differentiated by the fact that Red2+ gives check where as Rbd2+ does not. I would argue that Rd2+ is an illegal SAN move. Does xboard report with SAN? If so, that could be the issue. And I would blame crafty for it.

[hgm]

What Crafty sends is valid SAN. That only requires disambiguation between legal moves. So cutechess-cli is at fault here.

[Guenther]

What Crafty sends is valid SAN. That only requires disambiguation between legal moves. So cutechess-cli is at fault here.

I don't quite get this answer, or do I miss something? Both rook moves for Black are legal.

[Roland Chastain]

I don't quite get this answer, or do I miss something? Both rook moves for Black are legal.

In that case disambiguation comes from "+". I think this is the idea.

P.-S. But here is what the PGN specification says:

Neither the appearance nor the absence of either a check or checkmating indicator is used for disambiguation purposes. This means that if two (or more) pieces of the same type can move to the same square the differences in checking status of the moves does not allieviate the need for the standard rank and file disabiguation described above. (Note that a difference in checking status for the above may occur only in the case of a discovered check.)

[Guenther]

I don't quite get this answer, or do I miss something? Both rook moves for Black are legal.

In that case disambiguation comes from "+". I think this is the idea.

P.-S. But here is what the PGN specification says:

Neither the appearance nor the absence of either a check or checkmating indicator is used for disambiguation purposes. This means that if two (or more) pieces of the same type can move to the same square the differences in checking status of the moves does not allieviate the need for the standard rank and file disabiguation described above. (Note that a difference in checking status for the above may occur only in the case of a discovered check.)

Thanks, so a mere Rd2+ would be against the specification!
Note also that the pgn says 'Crafty 18.0', which is from the end of year 2000 or very early 2001!
Anyhow, it should be easy to just feed the game/fen into a GUI with log/debug feature for this old version
and check what Crafty really sends as move on this position.

[Guenther]

Case closed :)

Actually it obviously was a bug in a very old Crafty version, which was fixed in a later version.
See the output comparison for 18.1 and 25.6 (I had no 18.0 handy, but the bug was still in 18.1)
25.6 e.g. correctly sends Rexd2+ instead of Rxd2+.

Code: Select all

Crafty v18.1 (1 cpus)

White(1): xboard
White(1): setboard 1b6/6R1/8/5P2/p1b2k2/P7/1r1Br3/5K2 b - - 1 47

       +---+---+---+---+---+---+---+---+
    8  |   | *B|   |   |   |   |   |   |
       +---+---+---+---+---+---+---+---+
    7  |   |   |   |   |   |   | R |   |
       +---+---+---+---+---+---+---+---+
    6  |   |   |   |   |   |   |   |   |
       +---+---+---+---+---+---+---+---+
    5  |   |   |   |   |   | P |   |   |
       +---+---+---+---+---+---+---+---+
    4  | *P|   | *B|   |   | *K|   |   |
       +---+---+---+---+---+---+---+---+
    3  | P |   |   |   |   |   |   |   |
       +---+---+---+---+---+---+---+---+
    2  |   | *R|   | B | *R|   |   |   |
       +---+---+---+---+---+---+---+---+
    1  |   |   |   |   |   | K |   |   |
       +---+---+---+---+---+---+---+---+
         a   b   c   d   e   f   g   h

              puzzling over a move to ponder.
              clearing hash tables
         nss  depth   time  score   variation (1)
                3     0.00 -Mat03   1. ... Rxd2+ 2. Ke1 Re2+ 3. Kf1 Rb1#
                3->   0.00 -Mat03   1. ... Rxd2+ 2. Ke1 Re2+ 3. Kf1 Rb1#
                4     0.00 -Mat03   1. ... Rxd2+ 2. Ke1 Re2+ 3. Kf1 Rb1#
                4->   0.00 -Mat03   1. ... Rxd2+ 2. Ke1 Re2+ 3. Kf1 Rb1#
                5     0.00 -Mat03   1. ... Rxd2+ 2. Ke1 Re2+ 3. Kf1 Rb1#
         (3)    5->   0.05 -Mat03   1. ... Rxd2+ 2. Ke1 Re2+ 3. Kf1 Rb1#
Black(1): Rxd2+ [pondering]
              clearing hash tables
              time surplus   0.00  time limit 30.00 (3:30)
         nss  depth   time  score   variation (1)
                3     0.00 -Mat02   2. Ke1 Re2+ 3. Kf1 Rb1#
                4     0.00 -Mat02   2. Ke1 Re2+ 3. Kf1 Rb1#
                5     0.00 -Mat02   2. Ke1 Re2+ 3. Kf1 Rb1#
                6     0.03 -Mat02   2. Ke1 Re2+ 3. Kf1 Rb1#
                6->   0.03 -Mat02   2. Ke1 Re2+ 3. Kf1 Rb1#
                7     0.29 -Mat02   2. Ke1 Re2+ 3. Kf1 Rb1#
                7->   0.29 -Mat02   2. Ke1 Re2+ 3. Kf1 Rb1#
Black(1): go time 5000
              clearing hash tables
              time surplus   0.00  time limit 30.00 (3:30)
         nss  depth   time  score   variation (1)
                3     0.00 -Mat03   1. ... Rxd2+ 2. Ke1 Re2+ 3. Kf1 Rb1#
         (2)    3->   0.00 -Mat03   1. ... Rxd2+ 2. Ke1 Re2+ 3. Kf1 Rb1#
                4     0.00 -Mat03   1. ... Rxd2+ 2. Ke1 Re2+ 3. Kf1 Rb1#
                4->   0.00 -Mat03   1. ... Rxd2+ 2. Ke1 Re2+ 3. Kf1 Rb1#
                5     0.02 -Mat03   1. ... Rxd2+ 2. Ke1 Re2+ 3. Kf1 Rb1#
         (3)    5->   0.02 -Mat03   1. ... Rxd2+ 2. Ke1 Re2+ 3. Kf1 Rb1#
              time=0.02  cpu=100%  mat=7  n=46934  fh=98%  nps=10000
              ext-> checks=3539 recaps=12 pawns=57 1rep=367 thrt:71
              predicted=0  nodes=46934  evals=78
              endgame tablebase-> probes done=0  successful=0
              hashing-> trans/ref=53%  pawn=99%  used=5%
              SMP->  split=0  stop=0  data=0/64  cpu=0.02  elap=0.02

mate in 3 moves.

Black(1): Rxd2+
              time used:   0.02

Code: Select all

Crafty v25.6

machine has 4 processors

White(1): xboard

tellicsnoalias set 1 Crafty v25.6 (1 cpus)
tellicsnoalias kibitz Hello from Crafty v25.6! (1 cpus)
White(1): setboard 1b6/6R1/8/5P2/p1b2k2/P7/1r1Br3/5K2 b - - 1 47
Black(1): go time 5000
        time surplus   0.00  time limit 30.00 (2:30)
        depth     time       score   variation (1)
          9->   0.00/18.00  -Mat03   1. ... Rexd2+ 2. Ke1 Re2+ 3. Kf1 Rb1#
        time=0.00(100%)  nodes=22824(22.8K)  fh1=97%  pred=1  nps=2.3M
        chk=1.0K  qchk=2.2K  fp=9.1K  mcp=0  50move=1
        LMReductions:  1/210  2/155  3/125  4/25
        null-move (R):  3/262  4/39

mate in 3 moves.

Black(1): Rexd2+
              time used:   0.00

[jkominek]

Case closed

Actually it obviously was a bug in a very old Crafty version, which was fixed in a later version.
See the output comparison for 18.1 and 25.6 (I had no 18.0 handy, but the bug was still in 18.1)
25.6 e.g. correctly sends Rexd2+ instead of Rxd2+.

Good investigation. Thank you. The quote from specification seals it for all intents and purposes.

In contrast to v25.6, the final output from Crafty's log of the game under discussion has the move under-specified in its PV lines.

Code: Select all

White(47): time 1656
time remaining:  16.56 (crafty).
White(47): otim 0
time remaining:   0.00 (opponent).
White(47): b4d2
              time used:   0.04
              time limit 1.00 (5.14)
         nss  depth   time  score   variation (1)
                3     0.00 -Mat03   47. ... Rxd2+ 48. Ke1 Re2+ 49. Kf1
                                    Rb1#
                3->   0.00 -Mat03   47. ... Rxd2+ 48. Ke1 Re2+ 49. Kf1
                                    Rb1#
                4     0.01 -Mat03   47. ... Rxd2+ 48. Ke1 Re2+ 49. Kf1
                                    Rb1#
                4->   0.01 -Mat03   47. ... Rxd2+ 48. Ke1 Re2+ 49. Kf1
                                    Rb1#
                5     0.03 -Mat03   47. ... Rxd2+ 48. Ke1 Re2+ 49. Kf1
                                    Rb1#
         (3)    5->   0.22 -Mat03   47. ... Rxd2+ 48. Ke1 Re2+ 49. Kf1
                                    Rb1#
              time=0.22  cpu=100%  mat=7  n=154761  fh=99%  nps=703459
              ext-> checks=13251 recaps=11 pawns=1233 1rep=2430 thrt:12
              predicted=0  nodes=154761  evals=25
              endgame tablebase-> probes done=0  successful=0
              SMP->  split=0  stop=0  data=0/64  cpu=0.22  elap=0.22

mate in 3 moves.

Black(47): Rxd2+
              time used:   0.22
White(48): force
White(48): result 1-0 {Black makes an illegal move: Rxd2+}

Crafty 18.0 has a last modified date of 2001-Jan-08. The release notes advertise it as the first version to support xboard protocol version 2.

[jkominek]

As a follow-up note, it appears to me that Crafty, all the way up to the latest 25.6, has another SAN interpretation bug. It does not accept e1g1/e1c1 as castling moves (respectively for black). It only accepts "Big O" notation.

The note in https://www.chessprogramming.org/Algebr ... s_Notation says

XBoard
Per default, the Chess Engine Communication Protocol requires pure algebraic coordinate notation without from-to delimiters as input and output move format [1] .

The BNF for pure algebraic notation excludes the conventional O-O and O-O-O.

Is there a status setting in the xboard protocol that declares whether pure or extended notation is being supported?

[jkominek]

Note also that the pgn says 'Crafty 18.0', which is from the end of year 2000 or very early 2001!

Yet age appropriate for the computer it is running on, if you notice. It is Stockfish who is being transported back in time.