Crafty Stalemate detection bug

[Tony Thomas]

Not sure if Leo send the report to Robert, so I decided to post it here..

http://wbec-ridderkerk.forumotion.com/w ... -1-t29.htm

[bob]

Not sure if Leo send the report to Robert, so I decided to post it here..

http://wbec-ridderkerk.forumotion.com/w ... -1-t29.htm

I don't see any bug being found??? Can you tell me what is supposed to have happened that didn't, or what wasn't supposed to happen that did? I've not seen any stalemate detection problems in years, after literally millions of games...

[Tord Romstad]

Not sure if Leo send the report to Robert, so I decided to post it here..

http://wbec-ridderkerk.forumotion.com/w ... -1-t29.htm

I don't see any bug being found???

Huh? No bug being found? It looks like a fairly catastrophic bug to me!

Do the following positions (where Crafty claimed a stalemate, according to the link above) look like stalemates to you?

Tord

[d]8/5K2/p1q5/5k1p/8/8/8/8 w - - 6 70
[d]8/8/8/8/8/1k6/1r6/4K3 w - - 0 85
[d]7r/4k3/8/8/4BK2/4P3/8/8 w - - 0 90
[d]7Q/8/8/1R6/6K1/8/k6B/8 b - - 2 101
[d]8/r7/8/7K/2k4P/8/8/6q1 w - - 1 88
[d]8/8/8/6k1/8/4KPP1/8/R7 b - - 0 50
[d]8/8/8/8/8/5KP1/R6P/6k1 b - - 4 49
[d]3R4/7K/8/8/8/7k/8/7r w - - 38 73
[d]4q3/K4k2/1p6/8/P7/8/8/8 w - - 1 52
[d]4R2K/8/5p2/4p1k1/8/8/8/8 b - - 0 72
[d]7r/4k3/8/8/4BK2/4P3/8/8 w - - 0 90

[bob]

Not sure if Leo send the report to Robert, so I decided to post it here..

http://wbec-ridderkerk.forumotion.com/w ... -1-t29.htm

I don't see any bug being found???

Huh? No bug being found? It looks like a fairly catastrophic bug to me!

Do the following positions (where Crafty claimed a stalemate, according to the link above) look like stalemates to you?

Tord

Find any version of crafty that claims those positions are stalemates and I will agree with you. However, I tried each one and it offered correct evaluations for each. It is more likely the bug is in the interface somewhere, but without a log to see what Crafty actually sent to the GUI, debugging is not possible.

But, as I said, I specifically tested the draw by repetition, stalemate, insufficient material and 50-move-rule over a _bunch_ of games last year to verify that the new repetition code never failled. And then I looked at each game that ended by one of the above (final position and final eval by Crafty) to see if they agreed. The only cases I found where Crafty ran into a draw without expecting it was when Crafty thought it was worse, and the opponent chose to force a draw unexpectedly because the opponent thought it was worse as well...

This one seems to be a GUI problem of unknown origin until someone can show me a log file that might assist in determining how the apparent confusion is happening.

[d]8/5K2/p1q5/5k1p/8/8/8/8 w - - 6 70
[d]8/8/8/8/8/1k6/1r6/4K3 w - - 0 85
[d]7r/4k3/8/8/4BK2/4P3/8/8 w - - 0 90
[d]7Q/8/8/1R6/6K1/8/k6B/8 b - - 2 101
[d]8/r7/8/7K/2k4P/8/8/6q1 w - - 1 88
[d]8/8/8/6k1/8/4KPP1/8/R7 b - - 0 50
[d]8/8/8/8/8/5KP1/R6P/6k1 b - - 4 49
[d]3R4/7K/8/8/8/7k/8/7r w - - 38 73
[d]4q3/K4k2/1p6/8/P7/8/8/8 w - - 1 52
[d]4R2K/8/5p2/4p1k1/8/8/8/8 b - - 0 72
[d]7r/4k3/8/8/4BK2/4P3/8/8 w - - 0 90

And there is the reason I said "I see no problem in Crafty." With that many errors in that few games, would you not agree that in 300,000 games, I would have seen thousands of such cases rather than zero? Not to mention games on ICC where it averages playing about 500 games a week...

BTW, exactly how is Crafty able to "end the game" by claiming a draw??? I thought this had been discussed ad nauseum a while back, with the conclusion being that the GUI is supposed to decide when the game is over, not one of the two participants???

[Eelco de Groot]

What the positions all seemed to have in common is that they are all with 5 pieces or less, so I thought perhaps it was a tablebase problem, from some missing tablebases maybe? Does Crafty assign a draw if a tablebase can't be found, or if not maybe the GUI was responsible?

Regards, Eelco

[Tord Romstad]

This one seems to be a GUI problem of unknown origin until someone can show me a log file that might assist in determining how the apparent confusion is happening.

The GUI is almost certainly Winboard. Just ask Leo for the logs, I bet he usually keeps them for a while.

BTW, exactly how is Crafty able to "end the game" by claiming a draw??? I thought this had been discussed ad nauseum a while back, with the conclusion being that the GUI is supposed to decide when the game is over, not one of the two participants???

I'm confused. Don't XBoard/Winboard always accept draw claims from the engines?

Tord

[bob]

What the positions all seemed to have in common is that they are all with 5 pieces or less, so I thought perhaps it was a tablebase problem, from some missing tablebases maybe? Does Crafty assign a draw if a tablebase can't be found, or if not maybe the GUI was responsible?

Regards, Eelco

No. In fact, Crafty was the first program to correctly handle missing egtb files. For example, you have krpkr, but not kqrkr or krrkr. I have seen gui issues where Crafty can move _way_ too quickly for some GUIs and they seem to ignore the move if it comes in instantly.

[bob]

This one seems to be a GUI problem of unknown origin until someone can show me a log file that might assist in determining how the apparent confusion is happening.

The GUI is almost certainly Winboard. Just ask Leo for the logs, I bet he usually keeps them for a while.

BTW, exactly how is Crafty able to "end the game" by claiming a draw??? I thought this had been discussed ad nauseum a while back, with the conclusion being that the GUI is supposed to decide when the game is over, not one of the two participants???

I'm confused. Don't XBoard/Winboard always accept draw claims from the engines?

Tord

So far as I know, draw offers are passed across to the opponent, if the opponent accepts the game ends... In the case of play on ICC, ICC is the ultimate arbiter and winboard just passes the request/claim to it.

In my cluster testing, I always trust Crafty's claims in my referee program, but I have carefully checked the results any time we modify anything remotely related to draws or repetitions...

[Zach Wegner]

So far as I know, draw offers are passed across to the opponent, if the opponent accepts the game ends...

Draw _offers_. Not draw _claims_. Draw claims end the game instantly.

[Volker Pittlik]

...

Do the following positions (where Crafty claimed a stalemate, according to the link above) look like stalemates to you? ...

I did that for the first three positions. Environment: 64-bit Linux, unmodified xboard, tablebases on flashdrive, opponent: Glaurung.


[d]8/5K2/p1q5/5k1p/8/8/8/8 w - - 6 70


xboard debug:

Code: Select all

8606 >second: setboard 8/5K2/p1q5/5k1p/8/8/8/8 w - - 0 1
8607 >first : computer
8607 >first : name Glaurung 2.1
8607 >second: computer
8607 >second: name Crafty-22.1
8607 >first : time 30000
otim 30000
8607 >first : go
8607 <second: pong 1
8648 <first : tellicsnoalias kibitz Hello from Crafty v22.1! (1 cpus)
8648 <first : tellics resign
8648 <first : 0-1 {White resigns}

crafty log:

Code: Select all

White(1): setboard 8/5K2/p1q5/5k1p/8/8/8/8 w - - 0 1

       +---+---+---+---+---+---+---+---+
    8  |   | . |   | . |   | . |   | . |
       +---+---+---+---+---+---+---+---+
    7  | . |   | . |   | . |-K-| . |   |
       +---+---+---+---+---+---+---+---+
    6  |<P>| . |<Q>| . |   | . |   | . |
       +---+---+---+---+---+---+---+---+
    5  | . |   | . |   | . |<K>| . |<P>|
       +---+---+---+---+---+---+---+---+
    4  |   | . |   | . |   | . |   | . |
       +---+---+---+---+---+---+---+---+
    3  | . |   | . |   | . |   | . |   |
       +---+---+---+---+---+---+---+---+
    2  |   | . |   | . |   | . |   | . |
       +---+---+---+---+---+---+---+---+
    1  | . |   | . |   | . |   | . |   |
       +---+---+---+---+---+---+---+---+
         a   b   c   d   e   f   g   h

White(1): computer
playing a computer!
White(1): name Glaurung 2.1
Crafty 22.1 vs Glaurung
White(1): time 30000
time remaining:   5:00 (crafty).
White(1): otim 30000
time remaining:   5:00 (opponent).
White(1): go
              time limit 8.78 (+0.00) (1:21)
              depth   time  score   variation (1)
              time=0.01  mat=-11  n=27  fh=100%  nps=1.0M
              ext-> check=0 1rep=0 mate=0 pp=0 reduce=12/6
              predicted=0  evals=4  50move=0  EGTBprobes=4  hits=4
              SMP->  splits=0  aborts=0  data=0/128  elap=0.01

mated in 3 moves.

tellics resign
0-1 {White resigns}

[d]8/8/8/8/8/1k6/1r6/4K3 w - - 0 85

debug:

Code: Select all

329991 >second: setboard 8/8/8/8/8/1k6/1r6/4K3 w - - 0 1
...
330031 <first : tellicsnoalias kibitz Hello from Crafty v22.1! (1 cpus)
330031 <first : tellics resign
330031 <first : 0-1 {White resigns}

crafty log:

Code: Select all

White(1): setboard 8/8/8/8/8/1k6/1r6/4K3 w - - 0 1
...
White(1): go
              time limit 8.78 (+0.00) (1:21)
              depth   time  score   variation (1)
              time=0.00  mat=-5  n=15  fh=100%  nps=1.0M
              ext-> check=0 1rep=0 mate=0 pp=0 reduce=4/2
              predicted=0  evals=2  50move=0  EGTBprobes=2  hits=2
              SMP->  splits=0  aborts=0  data=0/128  elap=0.00

mated in 6 moves.

tellics resign
0-1 {White resigns}

[d]7r/4k3/8/8/4BK2/4P3/8/8 w - - 0 90


That _is_ a draw in 16. Replay was faster:

Code: Select all

[Event "Computer chess game"]
[Site "vpittlik"]
[Date "2008.06.12"]
[Round "-"]
[White "Glaurung 2.1"]
[Black "Crafty-22.1"]
[Result "1/2-1/2"]
[TimeControl "300+1"]
[FEN "7r/4k3/8/8/4BK2/4P3/8/8 w - - 0 1"]
[SetUp "1"]

{--------------
. . . . . . . r
. . . . k . . .
. . . . . . . .
. . . . . . . .
. . . . B K . .
. . . . P . . .
. . . . . . . .
. . . . . . . .
white to play
--------------}
1. Ke5 Rd8 2. Bd5 Rc8 3. e4 Rc1 4. Be6 Re1 5. Bg4 Rxe4+ 6. Kxe4
{Draw by insufficient material} 1/2-1/2

I couldn't reproduce these false draw claims. Possibly there was something wrong with version 22.0 (unlikely in my opinion) or with the draw detection tool?

Volker