stockfish's illusions

[Uri Blass]

This position could happen in my last game
When I analyzed it with stockfish I got the following strange analysis at small depths.
Stockfish believes at small depth that 24.Bf3 is good enough to win the queen inspite of the fact that the queen has 3 different safe squares(g5 f5 a5)

I wonder if all these moves are pruned by late move reduction at depth 3-6 or if there is a different reason that stockfish does not see it.

I do not plan to look at stockfish's code to find out but I guess that it may be possible to improve stockfish by better order of moves and if Qa5 Qf5 Qg5 are not searched first after Bf3 the order is bad.

[D]5rk1/pB2pp1p/3n2pb/7q/2PP4/6Q1/PB3P1P/4R1K1 w - - 1 24

Stockfish_15052905_x64_modern:
1/1 00:00 88 44k +1.01 24.Bd5
2/2 00:00 152 76k +2.68 24.Bd5 a6 25.Rxe7
3/3 00:00 328 164k +5.69 24.Bf3 Qxf3 25.Qxf3 Nxc4
4/4 00:00 403 202k +5.69 24.Bf3 Qxf3 25.Qxf3 Nxc4
5/5 00:00 908 76k +2.98 24.Bf3 Nf5 25.Bxh5 Nxg3 26.hxg3 gxh5
6/6 00:00 1k 69k +2.98 24.Bf3 Nf5 25.Bxh5 Nxg3 26.hxg3 gxh5
7/9 00:00 7k 346k +0.44 24.Bd5 Nf5 25.Qc7 Bg7 26.Qxa7 Qg5+ 27.Kh1 e6

[bob]

This position could happen in my last game
When I analyzed it with stockfish I got the following strange analysis at small depths.
Stockfish believes at small depth that 24.Bf3 is good enough to win the queen inspite of the fact that the queen has 3 different safe squares(g5 f5 a5)

I wonder if all these moves are pruned by late move reduction at depth 3-6 or if there is a different reason that stockfish does not see it.

I do not plan to look at stockfish's code to find out but I guess that it may be possible to improve stockfish by better order of moves and if Qa5 Qf5 Qg5 are not searched first after Bf3 the order is bad.

[D]5rk1/pB2pp1p/3n2pb/7q/2PP4/6Q1/PB3P1P/4R1K1 w - - 1 24

Stockfish_15052905_x64_modern:
1/1 00:00 88 44k +1.01 24.Bd5
2/2 00:00 152 76k +2.68 24.Bd5 a6 25.Rxe7
3/3 00:00 328 164k +5.69 24.Bf3 Qxf3 25.Qxf3 Nxc4
4/4 00:00 403 202k +5.69 24.Bf3 Qxf3 25.Qxf3 Nxc4
5/5 00:00 908 76k +2.98 24.Bf3 Nf5 25.Bxh5 Nxg3 26.hxg3 gxh5
6/6 00:00 1k 69k +2.98 24.Bf3 Nf5 25.Bxh5 Nxg3 26.hxg3 gxh5
7/9 00:00 7k 346k +0.44 24.Bd5 Nf5 25.Qc7 Bg7 26.Qxa7 Qg5+ 27.Kh1 e6

That can't be a bug. They are apparently doing some CRAZY pruning there. After Bf3, black has to move Q, and even if those moves are reduced, they don't lose the queen when dropping right in to q-search. This is likely late move pruning. Which probably means the good moves get ordered late rather than early here, whether by bad history counter or what I don't know.

I looked at a shallow dump of Crafty's tree. at depth=4, after Bf3 the first move it searches is a queen move from hash table. Then other queen moves follow quickly:

1 Bf3 d: 4 [ 0.97, 1.45] n:2347 SearchMoves(S:1) (t=0)
2 Qf5 d: 3 [ -1.45, -0.97] n:2348 SearchMoves(S:2) (t=0)
2 Nxc4 d: 3 [ -1.45, -0.97] n:2811 SearchMoves(S:4) (t=0)
2 Nf5 d: 3 [ -1.45, -0.97] n:2842 SearchMoves(S:5) (t=0)
2 Qa5 d: 3 [ -1.45, -0.97] n:2873 SearchMoves(S:10) (t=0)
2 Qb5 d: 3 [ -1.23, -0.97] n:3184 SearchMoves(S:10) (t=0)
2 Qc5 d: 3 [ -1.23, -0.97] n:3187 SearchMoves(S:10) (t=0)
2 f5 d: 3 [ -1.23, -0.97] n:3190 SearchMoves(S:10) (t=0)
2 e6 d: 3 [ -1.23, -0.97] n:3194 SearchMoves(S:10) (t=0)
2 f6 d: 3 [ -1.23, -0.97] n:3198 SearchMoves(S:10) (t=0)
2 Qxf3 d: 3 [ -1.23, -0.97] n:3202 SearchMoves(S:10) (t=0)
2 Qh4 d: 3 [ -1.23, -0.97] n:3231 SearchMoves(S:10) (t=0)
2 Ne4 d: 3 [ -1.23, -0.97] n:3234 SearchMoves(S:10) (t=0)
2 Nb5 d: 3 [ -1.23, -0.97] n:3238 SearchMoves(S:10) (t=0)
2 Nc8 d: 3 [ -1.23, -0.97] n:3241 SearchMoves(S:10) (t=0)
2 Ne8 d: 3 [ -1.23, -0.97] n:3244 SearchMoves(S:10) (t=0)
2 Qd5 d: 3 [ -1.23, -0.97] n:3247 SearchMoves(S:10) (t=0)
2 Qe5 d: 3 [ -1.23, -0.97] n:3249 SearchMoves(S:10) (t=0)
2 Qxh2+ d: 3 [ -1.23, -0.97] n:3252 SearchMoves(S:10) (t=0)
2 Nb7 d: 3 [ -1.23, -0.97] n:3281 SearchMoves(S:10) (t=0)
2 Bg5 d: 3 [ -1.23, -0.97] n:3284 SearchMoves(S:10) (t=0)
2 Bg7 d: 3 [ -1.23, -0.97] n:3288 SearchMoves(S:10) (t=0)
2 Ra8 d: 3 [ -1.23, -0.97] n:3292 SearchMoves(S:10) (t=0)
2 Rb8 d: 3 [ -1.23, -0.97] n:3296 SearchMoves(S:10) (t=0)
2 Rc8 d: 3 [ -1.23, -0.97] n:3301 SearchMoves(S:10) (t=0)
2 Rd8 d: 3 [ -1.23, -0.97] n:3306 SearchMoves(S:10) (t=0)
2 Re8 d: 3 [ -1.23, -0.97] n:3310 SearchMoves(S:10) (t=0)
2 Qh3 d: 3 [ -1.23, -0.97] n:3314 SearchMoves(S:10) (t=0)
2 Qg4 d: 3 [ -1.23, -0.97] n:3317 SearchMoves(S:10) (t=0)
2 Qg5 d: 3 [ -1.23, -0.97] n:3320 SearchMoves(S:10) (t=0)
2 Kg7 d: 3 [ -1.23, -0.97] n:3324 SearchMoves(S:10) (t=0)
2 Kh8 d: 3 [ -1.23, -0.97] n:3328 SearchMoves(S:10) (t=0)
2 a5 d: 3 [ -1.23, -0.97] n:3332 SearchMoves(S:10) (t=0)
2 e5 d: 3 [ -1.23, -0.97] n:3336 SearchMoves(S:10) (t=0)
2 g5 d: 3 [ -1.23, -0.97] n:3341 SearchMoves(S:10) (t=0)
2 a6 d: 3 [ -1.23, -0.97] n:3344 SearchMoves(S:10) (t=0)
2 Bf4 d: 3 [ -1.23, -0.97] n:3348 SearchMoves(S:10) (t=0)
2 Bc1 d: 3 [ -1.23, -0.97] n:3356 SearchMoves(S:10) (t=0)
2 Bd2 d: 3 [ -1.23, -0.97] n:3361 SearchMoves(S:10) (t=0)
2 Be3 d: 3 [ -1.23, -0.97] n:3366 SearchMoves(S:10) (t=0)
4 0.04/38.40 1.23 1. Bf3 Qa5 2. Rxe7 Nxc4

In the output above, near the end of each line, there is something like (S:n). S means serial search (not parallel at this ply, and in this test I only used one thread anyway). The "n" is the phase. Here 2 is hash move, 4 is non-losing captures, 5 is killer #1, and 10 is remaining moves (ordered by history counters).

No idea why they wouldn't have one queen move before they start pruning...

[Eelco de Groot]

The number of nodes at depth 6 is still very low, less than one kilonode here with this SfVerification version, no verification search yet because the depths are too low, so that is probbly exactly what you would get.

Stockfish 250515 by Tord Romstad, Marco Costalba and Joona Kiiski
position fen 5rk1/pB2pp1p/3n2pb/7q/2PP4/6Q1/PB3P1P/4R1K1 w - - 1 24
go depth 10
info depth 1 seldepth 1 multipv 1 score cp 99 nodes 88 nps 44000 tbhits 0 time 2
pv b7d5
info depth 2 seldepth 2 multipv 1 score cp 265 nodes 152 nps 6608 tbhits 0 time
23 pv b7d5 a7a6 e1e7
info depth 3 seldepth 3 multipv 1 score cp 567 nodes 326 nps 7581 tbhits 0 time
43 pv b7f3 h5f3 g3f3 d6c4
info depth 4 seldepth 4 multipv 1 score cp 567 nodes 401 nps 9325 tbhits 0 time
43 pv b7f3 h5f3 g3f3 d6c4
info depth 5 seldepth 5 multipv 1 score cp 298 nodes 898 nps 19955 tbhits 0 time
45 pv b7f3 d6f5 f3h5 f5g3 h2g3 g6h5
info depth 6 seldepth 6 multipv 1 score cp 298 nodes 1021 nps 22195 tbhits 0 tim
e 46 pv b7f3 d6f5 f3h5 f5g3 h2g3 g6h5
info depth 7 seldepth 7 multipv 1 score cp -88 nodes 3877 nps 58742 tbhits 0 tim
e 66 pv b7d5 d6f5 d5f3 f5g3 f3h5 g3h5 e1e7
info depth 8 seldepth 8 multipv 1 score cp -104 nodes 5565 nps 79500 tbhits 0 ti
me 70 pv b7d5 d6f5 d5f3 f5g3 f3h5 g3h5 e1e7 a7a6
info depth 9 seldepth 10 multipv 1 score cp -82 nodes 10429 nps 101252 tbhits 0
time 103 pv b7d5 d6f5 d5f3 f5g3 f3h5 g3h5 e1e7 a7a6 d4d5 f7f5
info depth 10 seldepth 14 multipv 1 score cp 0 nodes 23620 nps 160680 tbhits 0 t
ime 147 pv b7d5 h6d2 e1f1 d6f5 g3g2 h5h4 g2e4 h4h5 e4g2
bestmove b7d5 ponder h6d2

It is probably easy to fix if you switch off Late Move Pruning in PV nodes. Which Lucas will tell you is about a zero Elo change. There is no real benefit because these Q captures B nodes will have to be searched anyway even if you know the Queen can easily escape from depth seven, so from then you will be searching them with LMP on when it is just a side branch and having them searched without LMP before (in the improved version, after it fails high) is not much more than a little overhead. It (the improved version) only looks better in positions where the evasion is over the horizon, that would be at the end of a PV for instance. Or here at depth 3-6.

[Uri Blass]

The number of nodes at depth 6 is still very low, less than one kilonode here with this SfVerification version, no verification search yet because the depths are too low, so that is probbly exactly what you would get.

Stockfish 250515 by Tord Romstad, Marco Costalba and Joona Kiiski
position fen 5rk1/pB2pp1p/3n2pb/7q/2PP4/6Q1/PB3P1P/4R1K1 w - - 1 24
go depth 10
info depth 1 seldepth 1 multipv 1 score cp 99 nodes 88 nps 44000 tbhits 0 time 2
pv b7d5
info depth 2 seldepth 2 multipv 1 score cp 265 nodes 152 nps 6608 tbhits 0 time
23 pv b7d5 a7a6 e1e7
info depth 3 seldepth 3 multipv 1 score cp 567 nodes 326 nps 7581 tbhits 0 time
43 pv b7f3 h5f3 g3f3 d6c4
info depth 4 seldepth 4 multipv 1 score cp 567 nodes 401 nps 9325 tbhits 0 time
43 pv b7f3 h5f3 g3f3 d6c4
info depth 5 seldepth 5 multipv 1 score cp 298 nodes 898 nps 19955 tbhits 0 time
45 pv b7f3 d6f5 f3h5 f5g3 h2g3 g6h5
info depth 6 seldepth 6 multipv 1 score cp 298 nodes 1021 nps 22195 tbhits 0 tim
e 46 pv b7f3 d6f5 f3h5 f5g3 h2g3 g6h5
info depth 7 seldepth 7 multipv 1 score cp -88 nodes 3877 nps 58742 tbhits 0 tim
e 66 pv b7d5 d6f5 d5f3 f5g3 f3h5 g3h5 e1e7
info depth 8 seldepth 8 multipv 1 score cp -104 nodes 5565 nps 79500 tbhits 0 ti
me 70 pv b7d5 d6f5 d5f3 f5g3 f3h5 g3h5 e1e7 a7a6
info depth 9 seldepth 10 multipv 1 score cp -82 nodes 10429 nps 101252 tbhits 0
time 103 pv b7d5 d6f5 d5f3 f5g3 f3h5 g3h5 e1e7 a7a6 d4d5 f7f5
info depth 10 seldepth 14 multipv 1 score cp 0 nodes 23620 nps 160680 tbhits 0 t
ime 147 pv b7d5 h6d2 e1f1 d6f5 g3g2 h5h4 g2e4 h4h5 e4g2
bestmove b7d5 ponder h6d2

It is probably easy to fix if you switch off Late Move Pruning in PV nodes. Which Lucas will tell you is about a zero Elo change. There is no real benefit because these Q captures B nodes will have to be searched anyway even if you know the Queen can easily escape from depth seven, so from then you will be searching them with LMP on when it is just a side branch and having them searched without LMP before (in the improved version, after it fails high) is not much more than a little overhead. It (the improved version) only looks better in positions where the evasion is over the horizon, that would be at the end of a PV for instance. Or here at depth 3-6.

What about improving the order of moves in a way that stockfish detect safe squares for a piece that is under threat and try them first?

I do not think that there is no real benefit because
Bf3 can happen somewhere close to the leaves of the tree and stockfish may get wrong conclusions about the score about it and indirectly wrong conclusions about the right move

In the following position that I composed stockfish simply play
Nxh5 at small depths because it believes that Qxh5 is losing at depth 6-7
and the pv is Nxh5 Rb8 so switch off late move pruning at pv nodes does not solve the problem because I think that Now Bf3 is not a pv node at depth 6-7.


[D]5rk1/p3pp1p/3n2pb/3B1q1b/2PP1N2/6Q1/PB3P1P/4R1K1 w - - 0 24

Stockfish_15052905_x64_modern:
1/1 00:00 62 62k +2.88 24.Nxh5 Qxh5 25.Rxe7
2/2 00:00 130 130k +2.88 24.Nxh5 Qxh5
3/3 00:00 289 289k +2.10 24.Nxh5 Qxh5 25.Rxe7 Qd1+ 26.Kg2
4/6 00:00 671 336k +2.37 24.Ne6 Qf6 25.Nxf8 Kxf8
5/6 00:00 3k 135k +1.57 24.Bc1 Qb1 25.Nxh5 Bxc1 26.Nf6+ exf6 27.Qxd6
6/6 00:00 4k 164k +1.55 24.Nxh5 Rb8 25.Bc1 Bxc1 26.Rxe7 Qxh5
7/7 00:00 5k 183k +1.55 24.Nxh5 Rb8 25.Bc1 Bxc1 26.Rxe7 Qxh5 27.Bxf7+ Kg7
8/10 00:00 14k 438k +0.65 24.Nd3 e6 25.Bxe6 fxe6 26.Qxd6 Qxd3 27.Qxe6+ Kh8 28.d5+ Bg7
9/15 00:00 25k 662k +1.00 24.Bc1 Rb8 25.Nxg6 Qxg6 26.Qxg6+ hxg6 27.Bxh6 e6 28.c5 exd5 29.cxd6


After Bf3 maybe Nf5 can come earlier then Qg5 Qf5 Qa5 but I see no different move that can come earlier and using LMP for the second move that you search is not logical and hopefully stockfish does not do it.

It may be interesting to see all the nodes of the tree of stockfish at small depth to see what are the moves that it considers for black after Bf3

The tree is very small so it should be easy to print the tree at small depths
by changing the code.

[Joerg Oster]

The number of nodes at depth 6 is still very low, less than one kilonode here with this SfVerification version, no verification search yet because the depths are too low, so that is probbly exactly what you would get.

Stockfish 250515 by Tord Romstad, Marco Costalba and Joona Kiiski
position fen 5rk1/pB2pp1p/3n2pb/7q/2PP4/6Q1/PB3P1P/4R1K1 w - - 1 24
go depth 10
info depth 1 seldepth 1 multipv 1 score cp 99 nodes 88 nps 44000 tbhits 0 time 2
pv b7d5
info depth 2 seldepth 2 multipv 1 score cp 265 nodes 152 nps 6608 tbhits 0 time
23 pv b7d5 a7a6 e1e7
info depth 3 seldepth 3 multipv 1 score cp 567 nodes 326 nps 7581 tbhits 0 time
43 pv b7f3 h5f3 g3f3 d6c4
info depth 4 seldepth 4 multipv 1 score cp 567 nodes 401 nps 9325 tbhits 0 time
43 pv b7f3 h5f3 g3f3 d6c4
info depth 5 seldepth 5 multipv 1 score cp 298 nodes 898 nps 19955 tbhits 0 time
45 pv b7f3 d6f5 f3h5 f5g3 h2g3 g6h5
info depth 6 seldepth 6 multipv 1 score cp 298 nodes 1021 nps 22195 tbhits 0 tim
e 46 pv b7f3 d6f5 f3h5 f5g3 h2g3 g6h5
info depth 7 seldepth 7 multipv 1 score cp -88 nodes 3877 nps 58742 tbhits 0 tim
e 66 pv b7d5 d6f5 d5f3 f5g3 f3h5 g3h5 e1e7
info depth 8 seldepth 8 multipv 1 score cp -104 nodes 5565 nps 79500 tbhits 0 ti
me 70 pv b7d5 d6f5 d5f3 f5g3 f3h5 g3h5 e1e7 a7a6
info depth 9 seldepth 10 multipv 1 score cp -82 nodes 10429 nps 101252 tbhits 0
time 103 pv b7d5 d6f5 d5f3 f5g3 f3h5 g3h5 e1e7 a7a6 d4d5 f7f5
info depth 10 seldepth 14 multipv 1 score cp 0 nodes 23620 nps 160680 tbhits 0 t
ime 147 pv b7d5 h6d2 e1f1 d6f5 g3g2 h5h4 g2e4 h4h5 e4g2
bestmove b7d5 ponder h6d2

It is probably easy to fix if you switch off Late Move Pruning in PV nodes. Which Lucas will tell you is about a zero Elo change. There is no real benefit because these Q captures B nodes will have to be searched anyway even if you know the Queen can easily escape from depth seven, so from then you will be searching them with LMP on when it is just a side branch and having them searched without LMP before (in the improved version, after it fails high) is not much more than a little overhead. It (the improved version) only looks better in positions where the evasion is over the horizon, that would be at the end of a PV for instance. Or here at depth 3-6.

From a private version I'm occasionally working on:

Code: Select all

position fen 5rk1/pB2pp1p/3n2pb/7q/2PP4/6Q1/PB3P1P/4R1K1 w - - 1 24
go depth 10
info depth 1 seldepth 1 multipv 1 score cp 97 nodes 88 nps 22000 tbhits 0 time 4 pv b7d5
info depth 2 seldepth 2 multipv 1 score cp 60 nodes 373 nps 93250 tbhits 0 time 4 pv b7f3 h5a5
info depth 3 seldepth 3 multipv 1 score cp 60 nodes 1040 nps 208000 tbhits 0 time 5 pv b7d5 d6f5 g3g2
info depth 4 seldepth 4 multipv 1 score cp 30 nodes 1983 nps 396600 tbhits 0 time 5 pv b7f3 h5a5 e1e7 d6c4
info depth 5 seldepth 5 multipv 1 score cp 31 nodes 4372 nps 624571 tbhits 0 time 7 pv b7d5 d6f5 g3g2 h6d2 e1b1
info depth 6 seldepth 6 multipv 1 score cp 10 nodes 7781 nps 972625 tbhits 0 time 8 pv b7d5 d6f5 g3g2 h6d2 d5f3 h5h4
info depth 7 seldepth 7 multipv 1 score cp -4 nodes 20684 nps 1477428 tbhits 0 time 14 pv b7d5 h6d2 e1f1 d6f5 g3g2 f8b8 f1b1
info depth 8 seldepth 8 multipv 1 score cp -28 nodes 31250 nps 1736111 tbhits 0 time 18 pv b7d5 h6d2 e1f1 d6f5 g3g2 f8b8 d5b7 d2f4
info depth 9 seldepth 10 multipv 1 score cp -54 nodes 74271 nps 2063083 tbhits 0 time 36 pv b7d5 h6d2 e1f1 d6f5 g3g2 f8b8 b2a3 h5h4 a3c5 f5d4
info depth 10 seldepth 15 multipv 1 score cp -61 nodes 118132 nps 2147854 tbhits 0 time 55 pv b7d5 h6d2 e1a1 d6f5 g3g2 f8b8 b2a3 f5d4 a3e7 h5e5 a1d1
bestmove b7d5 ponder h6d2

For my taste SF is reducing/pruning too much in the first plies ...

[zullil]

Please see https://groups.google.com/forum/#!topic ... FfQ_sTL724

[syzygy]

What about improving the order of moves in a way that stockfish detect safe squares for a piece that is under threat and try them first?

I do not think that there is no real benefit because
Bf3 can happen somewhere close to the leaves of the tree and stockfish may get wrong conclusions about the score about it and indirectly wrong conclusions about the right move

As always this will be a trade-off between time (including programming time) and quality/node. Good luck with your efforts.

[hgm]

It can't be good pruning evasions in nodes where all moves so far failed low because they lose a valuable piece that is under attack.

[Eelco de Groot]

Well, you have to draw the line somewhere if you don't want to search full width. Do you do Queen evasions in q-search? I think Stockfish only does King evasions there. In the last six plies, the search from Stockfish is much like qsearch. This has been the case since Tord started Glaurung, maybe fifteen years now? I don't know exactly. Maybe it has become a bit more visible by not treating PV nodes differently with respect to LMP. There are still differences by not allowing nullmove in PV nodes and of course by searching with a window that is a bit larger than 1.

[Uri Blass]

Well, you have to draw the line somewhere if you don't want to search full width. Do you do Queen evasions in q-search? I think Stockfish only does King evasions there. In the last six plies, the search from Stockfish is much like qsearch. This has been the case since Tord started Glaurung, maybe fifteen years now? I don't know exactly. Maybe it has become a bit more visible by not treating PV nodes differently with respect to LMP. There are still differences by not allowing nullmove in PV nodes and of course by searching with a window that is a bit larger than 1.

There is a difference because in qsearch there is a do nothing option that you do not calculate after it so if your queen is under threat and you go to qsearch you do not evaluate it as a loss of the queen only because there is a bad capture.