Stockfish-1.5 Detects mate then changes mind?

[royb]

In this position:

1q1r3k/3P1pp1/ppBR1n1p/4Q2P/P4P2/8/5PK1/8 w - - bm Rxf6;

I let Stockfish 1.5 analyze with two threads and the tail end of the analysis window shows this:

19 +1.57 1.Rd1 Qxe5 2.fxe5 Nxh5 3.Kf3 Kg8 4.Rh1 g6 5.Rb1 Kf8 6.Rxb6 Ke7 7.Rxa6 Ng7 8.Ke3 Ne6 9.Rb6 Nc5 10.a5 Nxd7 (23.50)
20 +1.37 1.Rd1 Qxe5 2.fxe5 Nxh5 3.Kf3 Kg8 4.Rh1 g6 5.Rb1 f6 6.e6 Kf8 7.Rxb6 Ke7 8.Bd5 f5 9.Ke3 a5 10.Kd4 Nf4 (41.34)
20 M12 1.Kf3 Kg8 2.Qe7 Qc7 3.Kg2 a5 4.f3 Qb8 5.Rxf6 gxf6 6.Bd5 Qc7 7.Qxf7+ Kh8 8.Qxf6+ Kh7 9.Kg3 Rxd7 10.Qg6+ Kh8 11.Qxh6+ Rh7 12.Qf8# (63.09)
21 +1.77 1.Kf3 Kg8 2.Qe7 Qc7 3.Kg2 Kh8 4.Rxf6 gxf6 5.Qxf6+ Kg8 6.Qxh6 Rxd7 7.Qg5+ Kf8 8.Bxd7 Qxd7 9.h6 Qd4 10.Kg3 Qd3+ 11.Kh2 Qd4 12.Kh3 Qd3+ 13.Kh4 Qd4 14.h7 Qh8 15.Qf5 (73.47)

Strange that at ply 20 it thinks 1. Kf3 is a mate in 12 but then with more time and another ply it thinks that Kf3 only scores 1.77 pawns.

It seems that once it detects mate in 12, surely the only other moves that are better are moves that mate more quickly? I suspect the mate in 12 is not accurate but I'm going to sleep now and will be unable to check it until much later on.

I have no reason to doubt that the stated best move (Rxf6) is the best and suspect the mate in 12 announcement is in error. A bug in Stockfish 1.5 ??

Roy

[Tord Romstad]

I can't reproduce this (which is not unexpected, as you were using two threads), but the general rule is that you can never trust the fail high scores reported by Stockfish, not even as lower bounds. You have to wait until the fail high is resolved, usually at the next iteration. Perhaps we should just display the upper boundary of the window instead of the score returned by the search when a fail high at the root occurs.

Joona's aspiration window search is highly unorthodox and frequently displays weird behavior. The only reason I like it is that it works incredibly well when playing games.

By the way, Stockfish 1.5 is no different from 1.4 in this respect.

[Uri Blass]

stockfish with a single thread(32 bits) does not see a mate score

New game
1q1r3k/3P1pp1/ppBR1n1p/4Q2P/P4P2/8/5PK1/8 w - - 0 1

Analysis by Stockfish 1.5 JA:

1.Kg2-g3
± (1.37) Depth: 1 00:00:00
1.Kg2-g3 Nf6xh5+ 2.Kg3-h4
² (0.68) Depth: 2 00:00:00
1.Rd6-d1 Qb8xe5 2.f4xe5 Nf6xh5
± (1.33) Depth: 2 00:00:00
1.Rd6-d1 Qb8-a7 2.Qe5-e7
± (1.29) Depth: 3 00:00:00
1.Rd6-d1 Qb8-a7 2.Qe5-e7 Qa7-c7
± (1.29) Depth: 4 00:00:00
1.Rd6-d1 Qb8-a7 2.Qe5-e7 Qa7-c7 3.Kg2-h3 Nf6xh5
± (0.72) Depth: 5 00:00:00
1.Rd6-d3 Qb8-a7 2.Rd3-g3 Rd8xd7 3.Bc6xd7 Qa7-b7+ 4.Kg2-f1 Qb7xd7
± (0.92) Depth: 5 00:00:00
1.Rd6-d3 Qb8-a7 2.Rd3-g3 Rd8xd7 3.Bc6xd7 Qa7-b7+ 4.Kg2-f1 Qb7xd7
± (0.92) Depth: 6 00:00:00 10kN
1.Rd6-d3 Qb8-a7 2.Rd3-g3 Rd8xd7 3.Bc6xd7 Qa7-b7+ 4.Kg2-f1 Qb7xd7 5.Qe5-b8+ Kh8-h7 6.Qb8xb6 Qd7xa4
± (1.17) Depth: 7 00:00:00 15kN
1.Rd6-d3 Qb8-a7 2.Rd3-g3 Rd8xd7 3.Bc6xd7 Qa7-b7+ 4.Kg2-f1 Qb7xd7 5.Qe5-b8+ Kh8-h7 6.Qb8xb6 Qd7xa4
± (1.17) Depth: 8 00:00:00 22kN
1.Rd6-d3 Qb8-a7 2.Rd3-g3 Rd8xd7 3.Bc6xd7 Qa7-b7+ 4.Kg2-f1 Qb7xd7 5.Qe5-b8+ Kh8-h7 6.Qb8xb6 Qd7xa4
± (1.17) Depth: 9 00:00:01 37kN
1.Rd6-d3 Qb8-a7 2.Rd3-g3 Rd8xd7 3.Bc6xd7 Qa7-b7+ 4.Kg2-f1 Qb7xd7 5.Qe5-b8+ Kh8-h7 6.Qb8xb6 Qd7xa4
± (0.96) Depth: 10 00:00:01 68kN
1.Rd6-d3 Qb8-a7 2.Qe5-e7 Qa7-c7 3.Kg2-f1 Kh8-g8 4.Rd3-g3 Nf6xd7 5.Rg3-d3 Qc7xc6 6.Qe7xd8+ Nd7-f8
± (1.05) Depth: 11 00:00:01 205kN
1.Kg2-h3 Kh8-g8 2.Rd6-d3 Qb8-a7 3.Rd3-g3 Kg8-h7 4.Kh3-h4 Kh7-h8 5.Qe5-e7 Qa7-c7 6.Qe7xf7 Qc7xf4+ 7.Kh4-h3
+- (1.41) Depth: 11 00:00:01 464kN
1.Kg2-h3 Kh8-g8 2.Rd6-d3 Qb8-a7 3.Rd3-g3 Kg8-h7 4.Qe5-f5+ Kh7-h8 5.Rg3-e3 Qa7-b8 6.Kh3-h4 Kh8-g8 7.Re3-e7
+- (1.49) Depth: 12 00:00:03 587kN
1.Kg2-h3 Kh8-g8 2.Rd6-d3 Qb8xe5 3.f4xe5 Nf6xh5 4.Kh3-g4 g7-g6 5.Rd3-b3 f7-f5+ 6.Kg4-h4 Kg8-f7 7.Rb3xb6 Kf7-e6 8.Bc6-f3+ Ke6xd7 9.Rb6xg6
+- (1.49) Depth: 13 00:00:04 879kN
1.Kg2-h3 Kh8-g8 2.Rd6-d3 Qb8xe5 3.f4xe5 Nf6xh5 4.Kh3-g4 g7-g6 5.Rd3-b3 f7-f5+ 6.Kg4-h4 Kg8-f7 7.Rb3xb6 Nh5-f4 8.Rb6xa6 g6-g5+ 9.Kh4-g3 Kf7-e7
± (1.29) Depth: 14 00:00:06 1233kN
1.Rd6-d3 Qb8-a7 2.Kg2-f1 Kh8-h7 3.Rd3-e3 Rd8xd7 4.Qe5-f5+ Kh7-g8 5.Bc6xd7 Qa7xd7 6.Qf5xd7 Nf6xd7 7.Re3-e8+ Kg8-h7 8.Kf1-e2 Nd7-f6 9.Re8-e7 Nf6xh5 10.Re7xf7
+- (1.49) Depth: 14 00:00:06 2781kN
1.Rd6-d3 Qb8-a7 2.Kg2-f1 Kh8-h7 3.Rd3-e3 Rd8xd7 4.Qe5-f5+ Kh7-g8 5.Bc6xd7 Qa7xd7 6.Qf5xd7 Nf6xd7 7.Re3-e8+ Kg8-h7 8.Re8-e7 Nd7-c5 9.Re7xf7 Nc5xa4 10.Rf7-e7
± (1.33) Depth: 15 00:00:08 4076kN
1.Rd6-d3 Qb8-a7 2.Kg2-f1 Kh8-h7 3.Kf1-e2 b6-b5 4.a4xb5 a6xb5 5.Bc6xb5 Qa7-a2+ 6.Ke2-f1 Nf6-g4 7.Qe5-f5+ Kh7-g8 8.Qf5xg4 Qa2-b1+ 9.Kf1-e2 Qb1xb5 10.Ke2-e3 Kg8-f8
+- (1.53) Depth: 16 00:00:13 6624kN
1.Rd6-d3 Qb8-a7 2.Kg2-f1 Kh8-h7 3.Kf1-e2 b6-b5 4.a4xb5 a6xb5 5.Bc6xb5 Kh7-g8 6.Rd3-g3 Qa7-b7 7.f2-f3 Qb7-b6 8.Ke2-f1 Qb6-e6 9.Qe5xe6 f7xe6
+- (1.53) Depth: 17 00:00:19 10493kN
1.Rd6-d3 Qb8xe5 2.f4xe5 Nf6xh5 3.Kg2-f3 Kh8-g8 4.Rd3-b3 f7-f6 5.e5-e6 Kg8-f8 6.Rb3xb6 Kf8-e7 7.Bc6-d5 f6-f5 8.Kf3-e3 Nh5-f6 9.Ke3-d4 a6-a5 10.Kd4-e5 Nf6xd5 11.Ke5xd5
+- (1.65) Depth: 18 00:00:26 16144kN
1.Rd6-d3 Qb8xe5 2.f4xe5 Nf6xh5 3.Kg2-f3 Kh8-g8 4.Rd3-b3 f7-f6 5.e5-e6 Kg8-f8 6.Rb3xb6 Kf8-e7 7.Bc6-d5 f6-f5 8.Kf3-e3 Nh5-f6 9.Ke3-d4 a6-a5 10.f2-f4 h6-h5 11.Kd4-e5 Nf6xd5 12.Ke5xd5
+- (1.53) Depth: 19 00:00:44 29149kN
1.Kg2-f3 Kh8-g8 2.Qe5-e7 Qb8-c7 3.Kf3-e2 Kg8-h8 4.Rd6xf6 g7xf6 5.Qe7xf6+ Kh8-g8 6.Qf6xh6 Rd8xd7 7.Bc6xd7 Qc7xd7 8.Qh6xb6 Qd7xa4 9.Qb6-b8+ Kg8-h7 10.Qb8-e5 Qa4-b4 11.Qe5-f5+ Kh7-g7 12.Qf5-g5+ Kg7-h7 13.Ke2-e3 a6-a5 14.Qg5-f5+ Kh7-g7 15.Qf5-g5+
+- (1.69) Depth: 19 00:01:15 50878kN
1.Kg2-f3 Kh8-g8 2.Qe5-e7 Qb8-c7 3.Kf3-e2 Kg8-h8 4.Rd6xf6 g7xf6 5.Qe7xf6+ Kh8-g8 6.Qf6xh6 Rd8xd7 7.Qh6-g5+ Kg8-f8 8.Bc6xd7 Qc7xd7 9.h5-h6 Qd7-e6+ 10.Ke2-f1 Qe6-c4+ 11.Kf1-g2 Qc4-d4 12.Kg2-g3 b6-b5 13.a4xb5 Qd4-d3+ 14.Kg3-g4 a6xb5
+- (1.73) Depth: 20 00:01:37 65488kN
1.Kg2-f3 Kh8-g8 2.Qe5-e7 Qb8-c7 3.Kf3-e2 Kg8-h8 4.Rd6xf6 g7xf6 5.Qe7xf6+ Kh8-g8 6.Qf6xh6 Rd8xd7 7.Qh6-g5+ Kg8-f8 8.Bc6xd7 Qc7xd7 9.h5-h6 Qd7-e6+ 10.Ke2-f1 Qe6-c4+ 11.Kf1-g2 Qc4-d4 12.Kg2-g3 b6-b5 13.a4xb5 Qd4-d3+ 14.Kg3-g4 a6xb5 15.Qg5-g7+ Kf8-e7
+- (1.77) Depth: 21 00:02:24 98201kN
1.Kg2-f3 Kh8-g8 2.Qe5-e7 Qb8-c7 3.Kf3-e2 Kg8-h8 4.Rd6xf6 g7xf6 5.Qe7xf6+ Kh8-g8 6.Qf6xh6 Rd8xd7 7.Qh6-g5+ Kg8-f8 8.Bc6xd7 Qc7xd7 9.h5-h6 Qd7-e6+ 10.Ke2-f1 Qe6-c4+ 11.Kf1-g2 Qc4-d4 12.Kg2-g3 Qd4-d3+ 13.f2-f3 Qd3-d4 14.Qg5-e5 Qd4-g1+ 15.Kg3-h4 Qg1-h1+ 16.Kh4-g5 Qh1xf3
+- (1.65) Depth: 22 00:04:13 170286kN
1.Kg2-f3 Kh8-g8 2.Qe5-e7 Qb8-c7 3.Kf3-e2 Kg8-h8 4.Ke2-f1 Kh8-g8 5.Kf1-g2 Kg8-h8 6.Rd6xf6 g7xf6 7.Qe7xf6+ Kh8-g8 8.Qf6xh6 Rd8xd7 9.Qh6-g5+ Kg8-f8 10.Bc6xd7 Qc7xd7 11.h5-h6 Qd7-d4 12.Kg2-g3 Qd4-c3+ 13.Kg3-h4 Qc3-d4 14.f2-f3 b6-b5 15.a4xb5 a6xb5 16.Qg5xb5 Qd4xf4+
+- (1.73) Depth: 23 00:11:26 464275kN

(so k, 07.10.2009)

[Uri Blass]

I can't reproduce this (which is not unexpected, as you were using two threads), but the general rule is that you can never trust the fail high scores reported by Stockfish, not even as lower bounds. You have to wait until the fail high is resolved, usually at the next iteration. Perhaps we should just display the upper boundary of the window instead of the score returned by the search when a fail high at the root occurs.

Joona's aspiration window search is highly unorthodox and frequently displays weird behavior. The only reason I like it is that it works incredibly well when playing games.

By the way, Stockfish 1.5 is no different from 1.4 in this respect.

I wonder if it is not better to have a more realistic score.
You can have a function that translates fail high score and previous score to more realistic score and decide that the more realistic score is the score of the move.

If I understand correctly
The situation today is that stockfish may miss some good winning move that is not mate at depth 21 because it found some wrong mate in 12 score and it may need depth 22 for it.

[zamar]

I'm not sure that aspiration window is the guilty one in here.

When we have aspiration fail high with mate score, one played move (in this case Kf3) returned _mate_ score. This should never happen (even in fail highs) if there is no mate.

You definetily should be able to trust the upper/lower bound returned by aspiration search. So there _is_ a bug. Unfortunately I'm not able to reproduce it either.

[Tord Romstad]

If I understand correctly
The situation today is that stockfish may miss some good winning move that is not mate at depth 21 because it found some wrong mate in 12 score and it may need depth 22 for it.

Actually, when a new best move is found and the score is above the aspiration window, Stockfish doesn't try to resolve the fail high, but skips all the remaining moves at the current iteration, and starts searching the new best move at the next iteration. This is very radical, and seems so crazy that it obviously can't work. One of Joona's unique skills is coming up with such outrageous ideas, being brave enough to actually spend time implementing and testing them, and proving that they are tremendously effective in practice.

[Tord Romstad]

I'm not sure that aspiration window is the guilty one in here.

When we have aspiration fail high with mate score, one played move (in this case Kf3) returned _mate_ score. This should never happen (even in fail highs) if there is no mate.

You definetily should be able to trust the upper/lower bound returned by aspiration search. So there _is_ a bug. Unfortunately I'm not able to reproduce it either.

It shouldn't be that hard to find a reproducible example, I think. I have seen this happen numerous times. I'll have a look right now and see if I find something (but it's probably better to discuss by e-mail rather than making noise here on the CCC).

[zamar]

Joona's aspiration window search is highly unorthodox and frequently displays weird behavior. The only reason I like it is that it works incredibly well when playing games.

Do you have an example (preferably reproducible ) of such a weird behaviour? I want to underline that you must be able to trust the score (upper/lowerbound) returned by aspiration search. Any other behaviour I consider a serious bug.

[Uri Blass]

I'm not sure that aspiration window is the guilty one in here.

When we have aspiration fail high with mate score, one played move (in this case Kf3) returned _mate_ score. This should never happen (even in fail highs) if there is no mate.

You definetily should be able to trust the upper/lower bound returned by aspiration search. So there _is_ a bug. Unfortunately I'm not able to reproduce it either.

It shouldn't be that hard to find a reproducible example, I think. I have seen this happen numerous times. I'll have a look right now and see if I find something (but it's probably better to discuss by e-mail rather than making noise here on the CCC).

I think that this discussion is interesting and I see no problem with the noise here.

Other programmers may also learn things from the noise.
Maybe it is better to do it in the programming section and not in the general topics.

Uri

[Uri Blass]

If I understand correctly
The situation today is that stockfish may miss some good winning move that is not mate at depth 21 because it found some wrong mate in 12 score and it may need depth 22 for it.

Actually, when a new best move is found and the score is above the aspiration window, Stockfish doesn't try to resolve the fail high, but skips all the remaining moves at the current iteration, and starts searching the new best move at the next iteration. This is very radical, and seems so crazy that it obviously can't work. One of Joona's unique skills is coming up with such outrageous ideas, being brave enough to actually spend time implementing and testing them, and proving that they are tremendously effective in practice.

I wonder if you tried instead of skipping the rest of the moves to make a research with bigger depth after fail high(like you do after fail high in case of late move reductions)

There is going to be no difference in the score of the move that fail high but the rest of the moves are going to be searched with smaller depth.

In the next iteration you can simply skip the first move.

Note that skipping the first move in case that the branching factor was high in the last iteration may be also an interesting idea to try.
Here is an example from analysis by rybka.
The branching factor was very high at depth 24 because rybka needed a long time to get a score for Bd1 and rybka immediately get a score at depth 25 with the same pv so it seems simply to skip Bd1 at depth 25(this is from analysis of one of my correspondence games and I do not give the position and the pv).

23 2:06:44 690.267.702 92.954 +0.49 Be2d1
24 11:25:14 3.953.564,204 98.462 +0.52 Be2d1

25 11:25:14 3.953.564,224 98.462 +0.52 Be2d1

Uri