illogical eval from SF? BBKBPK +- 2.00 then BBKBK = 0.00

Discussion of anything and everything relating to chess playing software and machines.

Moderator: Ras

User avatar
hgm
Posts: 28361
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: illogical eval from SF? BBKBPK +- 2.00 then BBKBK = 0.00

Post by hgm »

Well, being useful to correspondence players doesn't give you any extra Elo, so it seems to be a non-negociable no-no for Stockfish...
syzygy
Posts: 5697
Joined: Tue Feb 28, 2012 11:56 pm

Re: illogical eval from SF? BBKBPK +- 2.00 then BBKBK = 0.00

Post by syzygy »

hgm wrote:
syzygy wrote:This is intentional, so not a bug.
Call it what you like, but that doesn't make it any less wrong.
Well, every engine has to make a choice on how to deal with repetitions.

Some engines only score a repetition as a draw if the position occurs for the 3rd time or if the position occurs for the 2nd time in the search tree. This is completely "correct". This is what my engine does.

Most engines each each repetition as a draw, so also if it occurs only once in the history of the root position and once in the search tree. This has the problem that if the opponent messes up in a won position, the engine might give him a second chance by moving back into this position thinking that it will draw. However, it has also advantages:
- draw by reps are detected earlier;
- the engine will not repeat positions if it thinks it is better.

My engine regularly repeats positions once while thinking it is ahead and only chooses another path the third time. This looks pretty bad, don't you think?
syzygy
Posts: 5697
Joined: Tue Feb 28, 2012 11:56 pm

Re: illogical eval from SF? BBKBPK +- 2.00 then BBKBK = 0.00

Post by syzygy »

phenri wrote:...do not tell me that you find it normal (even to the point of view of a developer or an end user), because as a result of repeated movement, the engine does not suggest the right move?
SF probably should not display Qf4, since it is assuming a draw after 88.Kd1. Displaying 88...Qf4 might be a bug.
This could have implications if interacting with table bases, and they suggest moves that could be repeated but Stockfish (or GUI, I do not know) refuses to show, it becomes problematic for by example a correspondance player or freestyle)
It has none of these implications.
Regarding correspondence play, it will only happen if the opponent plays a rather bad move in a good position and manages to find a better move on the 2nd try. That could happen yes, but if a correspondence player cannot handle this, I think he should look for a different hobby.
phenri
Posts: 284
Joined: Tue Aug 13, 2013 9:44 am

Re: illogical eval from SF? BBKBPK +- 2.00 then BBKBK = 0.00

Post by phenri »

syzygy wrote:
hgm wrote:
syzygy wrote:This is intentional, so not a bug.
Call it what you like, but that doesn't make it any less wrong.
Well, every engine has to make a choice on how to deal with repetitions.

Some engines only score a repetition as a draw if the position occurs for the 3rd time or if the position occurs for the 2nd time in the search tree. This is completely "correct". This is what my engine does.

Most engines each each repetition as a draw, so also if it occurs only once in the history of the root position and once in the search tree. This has the problem that if the opponent messes up in a won position, the engine might give him a second chance by moving back into this position thinking that it will draw. However, it has also advantages:
- draw by reps are detected earlier;
- the engine will not repeat positions if it thinks it is better.

My engine regularly repeats positions once while thinking it is ahead and only chooses another path the third time. This looks pretty bad, don't you think?
According to the rules of FIDE, 3 repetitions would be needed so that it is drawing, except here there are not 3 repetitions.

If I includes your reasoning, it is for you normal that the engine plays a move which belongs to the worst is possible be losing and at best all that is drawing, to avoid a repetition which is winning and in the worst drawing?

Well, apparently everyone who cares, especially those interested...
phenri
Posts: 284
Joined: Tue Aug 13, 2013 9:44 am

Re: illogical eval from SF? BBKBPK +- 2.00 then BBKBK = 0.00

Post by phenri »

syzygy wrote: SF probably should not display Qf4, since it is assuming a draw after 88.Kd1. Displaying 88...Qf4 might be a bug.
PV 88.Kd1 Qf4 is obviously a bug, except that few people recognize.
syzygy wrote:
This could have implications if interacting with table bases, and they suggest moves that could be repeated but Stockfish (or GUI, I do not know) refuses to show, it becomes problematic for by example a correspondance player or freestyle)
It has none of these implications.
Regarding correspondence play, it will only happen if the opponent plays a rather bad move in a good position and manages to find a better move on the 2nd try. That could happen yes, but if a correspondence player cannot handle this, I think he should look for a different hobby.
I was referring to cases of complicated positions finals and not easy to win when it is possible that the engine may suggest one or two repeated moves.
Uri Blass
Posts: 10825
Joined: Thu Mar 09, 2006 12:37 am
Location: Tel-Aviv Israel

Re: illogical eval from SF? BBKBPK +- 2.00 then BBKBK = 0.00

Post by Uri Blass »

phenri wrote:Just to illustrate my point ...

Image Image

...do not tell me that you find it normal (even to the point of view of a developer or an end user), because as a result of repeated movement, the engine does not suggest the right move?

This could have implications if interacting with table bases, and they suggest moves that could be repeated but Stockfish (or GUI, I do not know) refuses to show, it becomes problematic for by example a correspondance player or freestyle)
88...Kf4 in the pv is not normal but 88.Kd1 with 0.00 score is clearly normal and this is a behaviour of many engines.

I do not find it to be a problem in analysis because I know for many years that I cannot trust analysis of engines when they show a line that leads to repetition in the pv(assuming one of the previous moves was wrong) and need to give them the FEN without previous moves to get a correct analysis.
User avatar
hgm
Posts: 28361
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: illogical eval from SF? BBKBPK +- 2.00 then BBKBK = 0.00

Post by hgm »

syzygy wrote:My engine regularly repeats positions once while thinking it is ahead and only chooses another path the third time. This looks pretty bad, don't you think?
Yes, it does. But not as bad as not finding a checkmate.

It should also be possible to prevent this. You could score the first repetition of a game-history position not as a draw, but just scale its score (if it is not a mating score) towards a draw. This should appreciably reduce the chances for needless repetitions.

Another possibility is to make a distinction between moves the engine played, and moves that were forced by the user. The whole logic that the the engine would have found the win first time around if there was one sort of crumbles when the move was forced that first time. And when analyzing, moves are by definition all forced by the user.
Joerg Oster
Posts: 974
Joined: Fri Mar 10, 2006 4:29 pm
Location: Germany
Full name: Jörg Oster

Re: illogical eval from SF? BBKBPK +- 2.00 then BBKBK = 0.00

Post by Joerg Oster »

syzygy wrote:
hgm wrote:
syzygy wrote:This is intentional, so not a bug.
Call it what you like, but that doesn't make it any less wrong.
Well, every engine has to make a choice on how to deal with repetitions.

Some engines only score a repetition as a draw if the position occurs for the 3rd time or if the position occurs for the 2nd time in the search tree. This is completely "correct". This is what my engine does.

Most engines each each repetition as a draw, so also if it occurs only once in the history of the root position and once in the search tree. This has the problem that if the opponent messes up in a won position, the engine might give him a second chance by moving back into this position thinking that it will draw. However, it has also advantages:
- draw by reps are detected earlier;
- the engine will not repeat positions if it thinks it is better.

My engine regularly repeats positions once while thinking it is ahead and only chooses another path the third time. This looks pretty bad, don't you think?
Nothing wrong with that.
As long as it doesn't go for the 3-fold-rep, it even looks a bit human.
Many human players do this, for time-saving reasons, or for psycohological ones. The opponent thinks, oh great, he goes for the draw. And then, surprise, surprise... :D

But seriously, as long as the 3-fold-rep is being avoided, I wouldn't care much.
Better than
Jörg Oster
syzygy
Posts: 5697
Joined: Tue Feb 28, 2012 11:56 pm

Re: illogical eval from SF? BBKBPK +- 2.00 then BBKBK = 0.00

Post by syzygy »

hgm wrote:
syzygy wrote:My engine regularly repeats positions once while thinking it is ahead and only chooses another path the third time. This looks pretty bad, don't you think?
Yes, it does. But not as bad as not finding a checkmate.
In theory the "drawing move" might lead to the engine quickly being checkmated, but that can only happen if the opponent overlooked the checkmate the first time around. In engine-engine matches this is unlikely to happen (and this includes correspondence play). In engine-human matches it could happen but the chances are rather slim.
It should also be possible to prevent this. You could score the first repetition of a game-history position not as a draw, but just scale its score (if it is not a mating score) towards a draw. This should appreciably reduce the chances for needless repetitions.
I think the search would simply look for a PV not ending in a position that has been seen before. I doubt it would do much to prevent needless repetitions.

More importantly, it seems the choice of how to deal with this issue measurably affects Elo, the solution picked by SF and most other engines simply being the "best". The downside is mostly cosmetic.
Another possibility is to make a distinction between moves the engine played, and moves that were forced by the user. The whole logic that the the engine would have found the win first time around if there was one sort of crumbles when the move was forced that first time. And when analyzing, moves are by definition all forced by the user.
A special mode for analysis purposes would of course be possible.
syzygy
Posts: 5697
Joined: Tue Feb 28, 2012 11:56 pm

Re: illogical eval from SF? BBKBPK +- 2.00 then BBKBK = 0.00

Post by syzygy »

phenri wrote:According to the rules of FIDE, 3 repetitions would be needed so that it is drawing, except here there are not 3 repetitions.
Sure, but SF does not adjudicate the position as a draw. SF only scores the position as a draw. For any engine you can find zillions of positions for which the engine returns a score that does not conform to the game-theoretic outcome.
If I includes your reasoning, it is for you normal that the engine plays a move which belongs to the worst is possible be losing and at best all that is drawing, to avoid a repetition which is winning and in the worst drawing?
I find it hard to understand this sentence. In any event, it is normal for any engine to not always play the very best move.

The developers of SF have chosen to implement repetition detection in a way that overall most often results in the best move being played, i.e. the implementation that is measurably best in terms of Elo. The downside is a cosmetic one. I find it difficult to criticise the SF developers for not addressing a mostly cosmetic issue which would cost real Elo to solve.