Disabling Null Move Pruning in Stockfish

[bob]

If absolutely nobody but you can see the point in a particular test (such as those where you want to measure the effect of too low values of MAX_PLY... I don't see anyone protesting Marco's exclamation that those are "madness"), then maybe you should just consider the possibility that the others are right and that you might be wrong. And if you then still cannot see the problem, maybe just accept anyway that it is better to run such tests privately and not irritate everybody else.

I have nothing against your good ideas, but madness is madness.

I think the idea has merit and see the point in this test. I assume Dr. Hyatt does also, since he's devoting his cluster resources to trying it. Others have also expressed interest. Uri's not utilizing the stockfish test system resources for this idea so what's your problem? Your claim of 'madness' is also based on no data.

Note that I don't think it will work, but I am willing to run tests to measure the effect. The two things I need are (a) a suitable time control that is not nuts (40 moves in 2 hours is a bit much for 30K games), and (b) one or two depth limits to try, in addition to the default algorithm.

Trivial to run the test, it just burns computer cycles... And one never knows when something unusual will come out of such a test, I have been surprised in the past more than once where something I thought would work did not, and something I thought would not work actually did.

[syzygy]

If absolutely nobody but you can see the point in a particular test (such as those where you want to measure the effect of too low values of MAX_PLY... I don't see anyone protesting Marco's exclamation that those are "madness"), then maybe you should just consider the possibility that the others are right and that you might be wrong. And if you then still cannot see the problem, maybe just accept anyway that it is better to run such tests privately and not irritate everybody else.

I have nothing against your good ideas, but madness is madness.

I think the idea has merit and see the point in this test. I assume Dr. Hyatt does also, since he's devoting his cluster resources to trying it. Others have also expressed interest. Uri's not utilizing the stockfish test system resources for this idea so what's your problem? Your claim of 'madness' is also based on no data.

Madness refers to another test (somewhat remotely related to the Uri's nullmove programme). Uri is using the testing framework to find out at white time control a low MAX_PLY starts to hurt. Madness is not the word I used (Marco beat me to it).

There is no merit in finding out at what time control a MAX_PLY of 20 hurts Elo. And he actually is using the framework for that.

But to come back to the topic, my plain and simple question was: why not use your own hardware to do tests at 10 minutes per game. It will take a long time, but why is that a problem. Of course if Bob wants to do it for him, that's fine by me too. (But I note that Uri has already stated that he refuses to believe counterevidence.)

[Greg Strong]

Madness refers to another test (somewhat remotely related to the Uri's nullmove programme). Uri is using the testing framework to find out at white time control a low MAX_PLY starts to hurt. Madness is not the word I used (Marco beat me to it).

There is no merit in finding out at what time control a MAX_PLY of 20 hurts Elo. And he actually is using the framework for that.

Ok, I don't know anything about that...

But I note that Uri has already stated that he refuses to believe counterevidence.

Source? I've been reading Uri's comments for years so I very much doubt this.

[Uri Blass]

If absolutely nobody but you can see the point in a particular test (such as those where you want to measure the effect of too low values of MAX_PLY... I don't see anyone protesting Marco's exclamation that those are "madness"), then maybe you should just consider the possibility that the others are right and that you might be wrong. And if you then still cannot see the problem, maybe just accept anyway that it is better to run such tests privately and not irritate everybody else.

I have nothing against your good ideas, but madness is madness.

I think that calling it madness is insulting.
1)It make sense to ask me not to use too much time of stockfish framework time on it but I already limited the number of games to 40,000 and the time control to 20 seconds per game so it is not that I spend many hours of the framework on this test.

2)My tests are at low priority so they do not prevent other people to test.

3)I think that people who gave some productive patchs should have some rights about using the framework espacially when they do it in low priority

4)I think that the testing with small depths can be productive .
I understood from my testing with small depths that it is probably better to return draw value when the king is in check at max_ply and not to call the normal evaluation so I stopped all the tests that called the normal evaluation when the king is in check.

I think that by learning how many plies you need at fast time control it may be possible to get some conjecture about how many plies you need at long time control.

[Uri Blass]

If absolutely nobody but you can see the point in a particular test (such as those where you want to measure the effect of too low values of MAX_PLY... I don't see anyone protesting Marco's exclamation that those are "madness"), then maybe you should just consider the possibility that the others are right and that you might be wrong. And if you then still cannot see the problem, maybe just accept anyway that it is better to run such tests privately and not irritate everybody else.

I have nothing against your good ideas, but madness is madness.

I think the idea has merit and see the point in this test. I assume Dr. Hyatt does also, since he's devoting his cluster resources to trying it. Others have also expressed interest. Uri's not utilizing the stockfish test system resources for this idea so what's your problem? Your claim of 'madness' is also based on no data.

Note that I don't think it will work, but I am willing to run tests to measure the effect. The two things I need are (a) a suitable time control that is not nuts (40 moves in 2 hours is a bit much for 30K games), and (b) one or two depth limits to try, in addition to the default algorithm.

Trivial to run the test, it just burns computer cycles... And one never knows when something unusual will come out of such a test, I have been surprised in the past more than once where something I thought would work did not, and something I thought would not work actually did.

I suggest 5 minutes per game+5 seconds per move time control
depths limit to try 16 and 22

[Uri Blass]

Madness refers to another test (somewhat remotely related to the Uri's nullmove programme). Uri is using the testing framework to find out at white time control a low MAX_PLY starts to hurt. Madness is not the word I used (Marco beat me to it).

There is no merit in finding out at what time control a MAX_PLY of 20 hurts Elo. And he actually is using the framework for that.

Ok, I don't know anything about that...

But I note that Uri has already stated that he refuses to believe counterevidence.

Source? I've been reading Uri's comments for years so I very much doubt this.

I guess he means to the fact that I say that my theory is not refuted even if the test show bad results but it is not correct that I refuse to believe counter evidence.

Counter evidence only can prove that at the specific depth my idea does not work and I have no problem to believe it.
It does not prove that it does not work for depth that is big enough for time control that is long enough.

[syzygy]

But I note that Uri has already stated that he refuses to believe counterevidence.

Source? I've been reading Uri's comments for years so I very much doubt this.

Note only that tests can prove that my theory is right but cannot refute my theory because it is always possible that Crafty simply does not get depth that is big enough to earn elo from disabling null move pruning

[syzygy]

I think that by learning how many plies you need at fast time control it may be possible to get some conjecture about how many plies you need at long time control.

You could also just run a few very long searches on your machine on "normal" positions and measure the maximum number of recursions of the search. By "normal" I mean positions that could occur in a real game and not some contrived composition.

To measure the maximum number of recursions, you can simply insert a few statements in your local copy of SF's code. You do not have to campaign first for a change to how SF measures selective search depth. It is just a one-time experiment.

Note further that the reason you are looking at this has much more to do with the limit of the number of iterations in id_loop() (the condition ++depth <= MAX_PLY) than with the limit of recursions. Of course there is some relation between maximum depth and the maximum number of recursions, but I am sure the problem you are looking to solve has nothing to do with SF only doing 100 recursions and not 2000 recursions.

[Uri Blass]

But I note that Uri has already stated that he refuses to believe counterevidence.

Source? I've been reading Uri's comments for years so I very much doubt this.

Note only that tests can prove that my theory is right but cannot refute my theory because it is always possible that Crafty simply does not get depth that is big enough to earn elo from disabling null move pruning

I can explain the reason for my confidence that my theory is right and not because of hard max ply limit and not only because of zugzwang.

Let think about an extreme case.

Suppose that we have hardware that is strong enough to search to depth 1000.

You have 2 possibilities:
1)use null move pruning and be pratically blind in some positions(not only because of zugzwang but also because of long threats that verification search does not help to detect them).


because every ply can reduce 1/4 of the remaining depth+many plies that are dependent on the evaluation.

The problem is that the formula is
R>= 3 * ONE_PLY+ depth / 4

Imagine that the depth is 1000 plies when there is a winning plan
that is basically play regardless of the opponent moves
1)A1, 2)A2,3)A3,... 16)A16
and the program does not understand the position so it does not see the threat that is basically a plan of 16 moves unless it searches all of the moves.

I could expect it at least to find it at depth 1000 assuming the opponent cannot prevent the plan even if the program does not understand the position but see what can happen even without LMR

1)play move A1 search for a threat
remaining depth
1000-3-1000/4=747 plies
a reply of null move remaining depth 746 plies
2)Play move A2 search for a threat
remaining depth
746-3-746/4=557 plies
a reply of null move remaining depth 556 plies

Continue in this way and after A15 the program does not have depth that is big enough to see A16

2)disable null move pruning when the depth is very high.
You still are going to search big depth(slightly smaller but big depth) when the program is going to see the plan.

It is clear for me that the gain from additrional depth becomes smaller when the depth is bigger when I do not believe that there are no endgames that the program needs to find some plan of 16 moves when the static evaluation is not good enough to see a threat with less moves.

[Uri Blass]

I think that by learning how many plies you need at fast time control it may be possible to get some conjecture about how many plies you need at long time control.

You could also just run a few very long searches on your machine on "normal" positions and measure the maximum number of recursions of the search. By "normal" I mean positions that could occur in a real game and not some contrived composition.

To measure the maximum number of recursions, you can simply insert a few statements in your local copy of SF's code. You do not have to campaign first for a change to how SF measures selective search depth. It is just a one-time experiment.

Note further that the reason you are looking at this has much more to do with the limit of the number of iterations in id_loop() (the condition ++depth <= MAX_PLY) than with the limit of recursions. Of course there is some relation between maximum depth and the maximum number of recursions, but I am sure the problem you are looking to solve has nothing to do with SF only doing 100 recursions and not 2000 recursions.

I did not find positions from practical games when the maximal number of recursions was bigger than 100 but I found a case when it was 86 after some minutes.

Note that I did not analyze most of the positions from my games(I analyzed only something like 10 positions) and I did not analyze every position for many hours so I tend to believe that there are positions when stockfish reach the limit of 100 recursions after a long analysis so now it is better after marco increased the limit to 120.

Uri