Disabling Null Move Pruning in Stockfish

[Uri Blass]

Getting close to done.

Default is 5 better than using the depth<20 to trigger null-move, and 12 better than using depth<15.

Will post final numbers when test is finished. Still lots of competition for cluster nodes during the day.

Had forgotten to check. Here's the final results:

Code: Select all

    Crafty-24.0-1        2641    3    3 30080   58%  2578   32%
    Crafty-24.0R05-20    2638    3    3 30080   58%  2578   32%
    Crafty-24.0R05-15    2632    3    3 30080   57%  2578   33%

24.0-1 is normal Crafty,
24.0-R05-20 uses the depth < 20 limit to do a null-move search
24.0-R05-15 uses the depth < 15 limit

Not much different between default and 20, but didn't really expect much since 20 plies in blitz is limited compared to 15 plies or less.

I'm just a casual observer, but thanks for doing this.

So your suspicion here is that, had the time control allowed more searches to exceed depth 20, then Crafty-24.0R05-20 would have fared even worse against Crafty 24.0-1?

Yes. I don't know how much worse. But this is one of those ideas that cluster testing has problems with, because when you screw around with the search, things change at longer time controls and you would have no idea this idea had hurt if I had used 10s + 0.1s for the time control...

You also do not know if it is going to be worse
20 limit with 5+5 lose 3 elo when the statistical error is 3 elo so we are even not sure that it lose elo.

It may be interesting to have some distribution of the depth that Crafty gets in these 5+5 games in positions that it basically does null move pruning(and if it avoids null move pruning in pawn endgames then not include these positions)

I mean something like the following when the first number is the percentage of moves that you get depth that is bigger than the mentioned depth and the second number is the percentage of games that include at least one move with bigger depth.

depth >18 59.0% of the moves 99.9% of the games
depth >19 54.0% of the moves 99.1% of the games
depth >20 50.1% of the moves 97.0% of the games
depth >21 46.1% of the moves 93.0% of the games
depth >22 42% of the moves 84.6% of the games


If I see more than 10% of the games that have depth that is bigger than 25 then I think that it may be a good idea to test also 25 limit at 5+5 because even if it has no effect in most games it can help to detect zugzwangs in the minority of the games that you get depth that is bigger than 25

[Uri Blass]

Here is a test position from my correspondence game that is at least hard for Crafty23.8

[d]8/P1n1k3/8/7p/1p2K2p/7P/B7/8 w - - 7 64

Crafty has the following line in the pv
64.Ke5 Kd7 65.Bd5 Kc8 66.a8=Q+ Nxa8 that is a draw based on my understanding.

The right line is 64.Kf5 that is a winning move(64.Kf4 is also winning)
and I wonder if avoiding null move pruning can help Crafty to find the winning move
You can see the full game in the following site
http://www.iccf.com/game?id=335531

Edit:I find that default Crafty23.8 can find 64.Kf5 at depth 30 when I only give it the fen but it certainly has not enough time for it in blitz but note that the hard task is earlier.
Here is the position one ply earlier when Crafty seems to have no idea how to win
Ba2 is winning because of zugzwang(It is not the only winning move but Crafty does not see the win)

[d]8/P1n5/5k2/7p/1p2K2p/1B5P/8/8 w - - 5 63

[bob]

Getting close to done.

Default is 5 better than using the depth<20 to trigger null-move, and 12 better than using depth<15.

Will post final numbers when test is finished. Still lots of competition for cluster nodes during the day.

Had forgotten to check. Here's the final results:

Code: Select all

    Crafty-24.0-1        2641    3    3 30080   58%  2578   32%
    Crafty-24.0R05-20    2638    3    3 30080   58%  2578   32%
    Crafty-24.0R05-15    2632    3    3 30080   57%  2578   33%

24.0-1 is normal Crafty,
24.0-R05-20 uses the depth < 20 limit to do a null-move search
24.0-R05-15 uses the depth < 15 limit

Not much different between default and 20, but didn't really expect much since 20 plies in blitz is limited compared to 15 plies or less.

I'm just a casual observer, but thanks for doing this.

So your suspicion here is that, had the time control allowed more searches to exceed depth 20, then Crafty-24.0R05-20 would have fared even worse against Crafty 24.0-1?

Yes. I don't know how much worse. But this is one of those ideas that cluster testing has problems with, because when you screw around with the search, things change at longer time controls and you would have no idea this idea had hurt if I had used 10s + 0.1s for the time control...

You also do not know if it is going to be worse
20 limit with 5+5 lose 3 elo when the statistical error is 3 elo so we are even not sure that it lose elo.

It may be interesting to have some distribution of the depth that Crafty gets in these 5+5 games in positions that it basically does null move pruning(and if it avoids null move pruning in pawn endgames then not include these positions)

I mean something like the following when the first number is the percentage of moves that you get depth that is bigger than the mentioned depth and the second number is the percentage of games that include at least one move with bigger depth.

depth >18 59.0% of the moves 99.9% of the games
depth >19 54.0% of the moves 99.1% of the games
depth >20 50.1% of the moves 97.0% of the games
depth >21 46.1% of the moves 93.0% of the games
depth >22 42% of the moves 84.6% of the games


If I see more than 10% of the games that have depth that is bigger than 25 then I think that it may be a good idea to test also 25 limit at 5+5 because even if it has no effect in most games it can help to detect zugzwangs in the minority of the games that you get depth that is bigger than 25

Here is a test you CAN run. Very fast games. And slowly increase the ply at which null-move kicks in.. I don't do null-move at the root, ever, so starting the min ply at 1 would behave like the default. Increasing it to 2 would disable null move at ply=2 only, leaving at at all deeper plies. I could run this quickly, and will. Each additional ply of no-null-move is going to hurt. How much? I don't know exactly, but will by tomorrow morning. From a lot of testing, every ply you turn off null-move in costs you Elo. Zugzwang being irrelevant, because we know that zugzwang positions break null-move, but for non-zugzwang positions null-move is a winner. And non-zugzwang positions are WAY more frequent until you get to pawn-only endings where most don't do null move anyway.

[bob]

Here's the final results. the -1 means don't do null move unless ply > 1 (I never do null-move at ply=1 anyone so this is the default value. -2 means only do null at ply=3 and beyond.

Code: Select all

   Crafty-24.0R08-1     2641    4    4 30080   66%  2512   22%
   Crafty-24.0R08-2     2635    4    4 30080   65%  2512   22%
   Crafty-24.0R08-3     2634    4    4 30080   65%  2512   22%
   Crafty-24.0R08-4     2631    4    4 30080   65%  2512   22%
   Crafty-24.0R08-5     2615    4    4 30080   63%  2512   22%
   Crafty-24.0R08-6     2605    4    4 30080   62%  2512   23%
   Crafty-24.0R08-7     2596    4    4 30080   61%  2512   23%
   Crafty-24.0R08-8     2586    4    4 30080   59%  2512   23%

Any trickery costs Elo it seems. The more you go for zugzwang importance, the weaker overall (although the better it will do in zugzwang positions).

[xmas79]

what if we disable null move at different drafts? The idea is simple: suppose you disable nulls at draft 1, so the last ply Y=X-1 of depth X doesn't get nulls. That means if this node is an all node, then the DONT_TRY_NULL_MOVE flag will be stored in the hash. In the next iteration we will retrieve this information and at ply Z-2 where Z=X+1 we will skip null. The idea is let iteration framework build this table iteratively. I don't have any cluster to test seriously this idea (if it's worth!). See which draft (suppose 2 4 8 16) performs better.

Natale.

[Evert]

what if we disable null move at different drafts? The idea is simple: suppose you disable nulls at draft 1, so the last ply Y=X-1 of depth X doesn't get nulls. That means if this node is an all node, then the DONT_TRY_NULL_MOVE flag will be stored in the hash. In the next iteration we will retrieve this information and at ply Z-2 where Z=X+1 we will skip null. The idea is let iteration framework build this table iteratively. I don't have any cluster to test seriously this idea (if it's worth!). See which draft (suppose 2 4 8 16) performs better.

The question is really why you would expect that to work.

Null-move is a sound idea, if you are not in a zugzwang position. Statistically you're unlikely to be in one, therefore it makes sense, from the point of view of improving playing strength, to always try null-moves (from an analysis point of view this may bevery different, but that goes for other things too, for instance finding the shortest win rather than some random winning move). Clearly this leads to worse play in zugzwang positions, so you gain strength if you could detect that the position has a zugzwang and you disable null move.

Thing is: the only reliable way to detect a zugzwang is for the null-move to fail high, while none of the "real" moves do (I think of this as "there is no tempo move"). Any other method that has been tried (someone correct me if I'm wrong) gets too many false positives and switches off null-move unnecessarily. That hurts more than it gives.

So rather than randomly disable null-move based only on things like the remaining depth, I think it's more constructive to try to come up with an efficient criterion for detecting when not to apply null-move. This is of course also harder to do.
Perhaps a depth-dependent idea that would work is to switch from null-move pruning (at small remaining depth) to null-move reductions (at larger remaining depth), to verified null-move (only apply reduction if the verification search confirms that the null-move is probably sound; close to the root). However, I would think that someone already tried that...

[Uri Blass]

what if we disable null move at different drafts? The idea is simple: suppose you disable nulls at draft 1, so the last ply Y=X-1 of depth X doesn't get nulls. That means if this node is an all node, then the DONT_TRY_NULL_MOVE flag will be stored in the hash. In the next iteration we will retrieve this information and at ply Z-2 where Z=X+1 we will skip null. The idea is let iteration framework build this table iteratively. I don't have any cluster to test seriously this idea (if it's worth!). See which draft (suppose 2 4 8 16) performs better.

The question is really why you would expect that to work.

Null-move is a sound idea, if you are not in a zugzwang position. Statistically you're unlikely to be in one, therefore it makes sense, from the point of view of improving playing strength, to always try null-moves (from an analysis point of view this may bevery different, but that goes for other things too, for instance finding the shortest win rather than some random winning move). Clearly this leads to worse play in zugzwang positions, so you gain strength if you could detect that the position has a zugzwang and you disable null move.

Thing is: the only reliable way to detect a zugzwang is for the null-move to fail high, while none of the "real" moves do (I think of this as "there is no tempo move"). Any other method that has been tried (someone correct me if I'm wrong) gets too many false positives and switches off null-move unnecessarily. That hurts more than it gives.

So rather than randomly disable null-move based only on things like the remaining depth, I think it's more constructive to try to come up with an efficient criterion for detecting when not to apply null-move. This is of course also harder to do.
Perhaps a depth-dependent idea that would work is to switch from null-move pruning (at small remaining depth) to null-move reductions (at larger remaining depth), to verified null-move (only apply reduction if the verification search confirms that the null-move is probably sound; close to the root). However, I would think that someone already tried that...

I do not think that zugzwang is the only problem of very aggressive null move pruning that stockfish is using.
Even with no zugzwang finding long plans is a big problem in case that you do not detect by evaluation that the plan is good and I believe that there are cases that long plans work in chess including the middle game and I cannot expect the evaluation to detect everything.

Maybe it is better not to use recursive null move pruning or at least to limit the number of null moves that you allow in a line.

[zullil]

I do not think that zugzwang is the only problem of very aggressive null move pruning that stockfish is using.
Even with no zugzwang finding long plans is a big problem

Do you have an example of such a position, where zugzwang is not the reason that Stockfish fails?

[Uri Blass]

I do not think that zugzwang is the only problem of very aggressive null move pruning that stockfish is using.
Even with no zugzwang finding long plans is a big problem

Do you have an example of such a position, where zugzwang is not the reason that Stockfish fails?

I do not have a clear example because it is hard to prove that a long plan works but here is a game when I felt that black chances to win may involve some long plan of bringing the king to the second side of the board to support the a pawn as a passed pawn

http://www.iccf.com/game?id=417221

Here is the game
Chess programs did not like my 27...cxb4 move and prefered 27...b6 when after 28.b5 I see no chance to create a passed pawn so probably no win but I may be wrong.

[Event "MT-Hybl/A (CZE)"]
[Site "ICCF"]
[Date "2012.12.1"]
[Round "-"]
[White "Papenin, Nikolai"]
[Black "Blass, Uri"]
[Result "0-1"]
[WhiteElo "2707"]
[BlackElo "2602"]

1.g3 e5 2.Bg2 d5 3.d3 Nf6 4.Nf3 Bd6 5.c4 d4
6.O-O O-O 7.a3 c5 8.e3 Nc6 9.exd4 exd4 10.Bg5 h6
11.Bxf6 Qxf6 12.Nfd2 Qg6 13.Ne4 Be7 14.f4 Bd7 15.Nbd2 f5
16.Nf2 Rae8 17.Nf3 Kh7 18.Qd2 Qf6 19.Rae1 Bd6 20.Rxe8 Rxe8
21.Re1 Rxe1+ 22.Qxe1 g5 23.fxg5 hxg5 24.Qd2 g4 25.Nh4 Kg7
26.Bxc6 Bxc6 27.b4 cxb4 28.axb4 a6 29.c5 Bc7 0-1

[syzygy]

Here is the game

Is there any relation with nullmove?

Did SF like your move after you disabled nullmove?
Or do you just blame any perceived shortcoming of SF on nullmove?