Crafty 23.1 50-Move-Rule Draw Bug

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Crafty 23.1 50-Move-Rule Draw Bug

Post by bob »

hgm wrote:
jhaglund wrote:I don't think it matters. Player or computer, I'd still offer a draw playing a game.

It's only polite to offer a draw in this endgame. Who wants to move 50 more times? Any FIDE player will accept a draw or just blunder 1 of the pieces too force a draw from lack of mating material...
Offering is not the same as claiming. As it is, Crafty forfeits games by this, and I would say that does matter. If it wants to offer a draw, it should offer it, not one-sidedly terminate the game.

On top of this, there are situations where there is a forced mate that ends in KBKN, because the opponent has to recapture a piece (converting to KBKN) after which he is mated. I have not tested if Crafty makes an exception for this case, but engines that unconditionally declare draw as soon as the material is KBKN will declare draw in some won or lost positions.
You said "games". I just scanned 5M games and did not find a single instance where a game ended in KB vs KN (I assume you are refering to the case where the king is blocked by its own B so that the king can't move? IE BKh8, BBh7, WKf8, WNf7?

I personally do not believe this _ever_ happens in a real game. Why would black play Bh7 (helpmate)? In light of that, claiming a draw is actually not unreasonable. But it is easy enough to have it continue to play in such ridiculous positions, and just offer draws instead. But since most programs don't seem to accept draws...
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Crafty 23.1 50-Move-Rule Draw Bug

Post by bob »

Sven Schüle wrote:
bob wrote:Thanks, will investigate. There is an exception to the 50 move rule, in that the position is not drawn if one side is mated, regardless of how many reversible moves have been made. This is probably not handled correctly based on this game...
I can't see such an exception in the 23.1 code. In Search(), line 68, you return with a draw score if the function RepetitionCheck() has returned non-zero. This happens before searching all moves. RepetitionCheck() also handles the 50 moves draw and returns 1 if the 50 moves counter is >= 100. Since mate detection occurs later within Search(), this implementation can't detect the case shown here IMO. Please correct me if I have misunderstood your code.

I propose to separate repetition detection from 50 moves detection by moving the 50-moves related code from RepetitionCheck() into a new function which is then called right below the mate detection code at the bottom of Search(). There should be almost zero performance impact.

Sven
There isn't such an exception. I am working on one, where I will return drawscore if it is a repetition, but set a flag otherwise and check that at the bottom of search where it normally returns a score. If it returns a mate on the move score, it will do so immediately, otherwise it will check the 50 move flag and return draw if not delivering mate on the move. Easy enough and not so much overhead.
Sven
Posts: 4052
Joined: Thu May 15, 2008 9:57 pm
Location: Berlin, Germany
Full name: Sven Schüle

Re: Crafty 23.1 50-Move-Rule Draw Bug

Post by Sven »

bob wrote:There isn't such an exception. I am working on one, where I will return drawscore if it is a repetition, but set a flag otherwise and check that at the bottom of search where it normally returns a score. If it returns a mate on the move score, it will do so immediately, otherwise it will check the 50 move flag and return draw if not delivering mate on the move. Easy enough and not so much overhead.
This is essentially the same as I proposed since you effectively move the reaction to detecting 50 moves draw to the bottom of the search. Using a flag is probably an optimization which sounds good. But in this case you have two additional options in case your RepetitionCheck() function has set the 50 moves flag:

1) If you already know at that point that the moving side is not in check then you can return drawscore immediately.

2) Otherwise (if you don't know, or you know the moving side is in check), you can limit beta to drawscore since the current node is definitely an end-of-game position which is either "checkmated" or "draw by 50 moves" but never better.

Sven
User avatar
Zach Wegner
Posts: 1922
Joined: Thu Mar 09, 2006 12:51 am
Location: Earth

Re: Crafty 23.1 50-Move-Rule Draw Bug

Post by Zach Wegner »

Sven Schüle wrote:I propose to separate repetition detection from 50 moves detection by moving the 50-moves related code from RepetitionCheck() into a new function which is then called right below the mate detection code at the bottom of Search(). There should be almost zero performance impact.
Except the overhead of searching the whole subtree when it's actually a draw. Which I guess is still just about zero. :) If you're concerned about the cost, just check for mate directly when the fifty move counter is >= 100. You know you're going to return either draw or mate, so going through the whole search routine, generating all moves and searching them, etc. is just a waste of time. A quick mate check is always going to be faster, easier, and clearer.
User avatar
hgm
Posts: 27796
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: Crafty 23.1 50-Move-Rule Draw Bug

Post by hgm »

bob wrote:Why would black play Bh7 (helpmate)?
He might play it because it is actually Bxh7, and there was a Queen on h7.

[d] 6rK/4n3/7k/8/8/1B6/8/8 w - - 0 1

Here Bxg8 does not seem such a strange move. Especially since it s the only legal move... :lol:
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Crafty 23.1 50-Move-Rule Draw Bug

Post by bob »

hgm wrote:
bob wrote:Why would black play Bh7 (helpmate)?
He might play it because it is actually Bxh7, and there was a Queen on h7.

[d] 6rK/4n3/7k/8/8/1B6/8/8 w - - 0 1

Here Bxg8 does not seem such a strange move. Especially since it s the only legal move... :lol:
How often does that occur in real games? There were less than 10 such endings in total, and none of them resulted in an invalid draw claim. This is a _really_ rare occurrence. And the 50-move case is also exceedingly rare. Could not find a one of those where a mate on the last move was missed and the referee program logged a draw claim from Crafty yet it was either mated or mating on the move.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Crafty 23.1 50-Move-Rule Draw Bug

Post by bob »

Sven Schüle wrote:
bob wrote:There isn't such an exception. I am working on one, where I will return drawscore if it is a repetition, but set a flag otherwise and check that at the bottom of search where it normally returns a score. If it returns a mate on the move score, it will do so immediately, otherwise it will check the 50 move flag and return draw if not delivering mate on the move. Easy enough and not so much overhead.
This is essentially the same as I proposed since you effectively move the reaction to detecting 50 moves draw to the bottom of the search. Using a flag is probably an optimization which sounds good. But in this case you have two additional options in case your RepetitionCheck() function has set the 50 moves flag:

1) If you already know at that point that the moving side is not in check then you can return drawscore immediately.

2) Otherwise (if you don't know, or you know the moving side is in check), you can limit beta to drawscore since the current node is definitely an end-of-game position which is either "checkmated" or "draw by 50 moves" but never better.

Sven
What you suggested is exactly what I have done so far. I still have a glitch in the test at the bottom, as there is a "leak" thanks to the parallel search stuff that I have to handle as well, since it is possible to split on such a position and it gets handled differently.
User avatar
hgm
Posts: 27796
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: Crafty 23.1 50-Move-Rule Draw Bug

Post by hgm »

bob wrote:How often does that occur in real games? There were less than 10 such endings in total, and none of them resulted in an invalid draw claim. This is a _really_ rare occurrence. And the 50-move case is also exceedingly rare. Could not find a one of those where a mate on the last move was missed and the referee program logged a draw claim from Crafty yet it was either mated or mating on the move.
I have seen one posting in this forum in the past few years where a case exactly like this was discussed. The human player was mating the engine opponent on an ICS, and te ICS declared the game draw before he could perform the mate in one. (ICC suffers from the same bug as Crafty, or, if you want, consciously violates FIDE rules.)

I have seen one other case where an engine (Arasan) was checkmated on the 50th move in ChessWar. Engines that have this bug invite this this, no matter how rarely it wuld occur with bug-free engines: they start to play random moves when approaching the 50-move limit, because they think it does not matter anymore. And in KQxKQy-type end-games, where one side is constantly checking and you have no real cover, playing random evasions in the end might very well be fatal. As Arasan found out.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Crafty 23.1 50-Move-Rule Draw Bug

Post by bob »

hgm wrote:
bob wrote:How often does that occur in real games? There were less than 10 such endings in total, and none of them resulted in an invalid draw claim. This is a _really_ rare occurrence. And the 50-move case is also exceedingly rare. Could not find a one of those where a mate on the last move was missed and the referee program logged a draw claim from Crafty yet it was either mated or mating on the move.
I have seen one posting in this forum in the past few years where a case exactly like this was discussed. The human player was mating the engine opponent on an ICS, and te ICS declared the game draw before he could perform the mate in one. (ICC suffers from the same bug as Crafty, or, if you want, consciously violates FIDE rules.)

I have seen one other case where an engine (Arasan) was checkmated on the 50th move in ChessWar. Engines that have this bug invite this this, no matter how rarely it wuld occur with bug-free engines: they start to play random moves when approaching the 50-move limit, because they think it does not matter anymore. And in KQxKQy-type end-games, where one side is constantly checking and you have no real cover, playing random evasions in the end might very well be fatal. As Arasan found out.
I have fixed this. I was really talking about kb vs kn drawn due to insufficient material, where one side can actually win. I've never seen that kind of position in real games. Perhaps a moron vs a good player might result in that, I don't know, but one has to consciously self-mate oneself.

I will likely release 23.2 as soon as I can test it thoroughly to make sure nothing was broken, but it will no longer claim draws except for bare kings or king vs king and one minor piece. Should put this to rest.
BBauer
Posts: 658
Joined: Wed Mar 08, 2006 8:58 pm

Re: Crafty 23.1 50-Move-Rule Draw Bug

Post by BBauer »

Hi Bob.
The *rare* argument is not a good one.
Most positions in chess are *rare*.
IMHO a program should handle alle possible positions correct.
Such a program will I call a chess program.
In the past we have seen a lot of crap like cannot handle
- insufficient material
- underpromotion
- studies
Recently Kramnik said about a game Carlsen-Kramnik that Carlsen began to produce studies and mate threats.
kind regards
Bernhard