Disabling Null Move Pruning in Stockfish

[bob]

If absolutely nobody but you can see the point in a particular test (such as those where you want to measure the effect of too low values of MAX_PLY... I don't see anyone protesting Marco's exclamation that those are "madness"), then maybe you should just consider the possibility that the others are right and that you might be wrong. And if you then still cannot see the problem, maybe just accept anyway that it is better to run such tests privately and not irritate everybody else.

I have nothing against your good ideas, but madness is madness.

I think the idea has merit and see the point in this test. I assume Dr. Hyatt does also, since he's devoting his cluster resources to trying it. Others have also expressed interest. Uri's not utilizing the stockfish test system resources for this idea so what's your problem? Your claim of 'madness' is also based on no data.

Note that I don't think it will work, but I am willing to run tests to measure the effect. The two things I need are (a) a suitable time control that is not nuts (40 moves in 2 hours is a bit much for 30K games), and (b) one or two depth limits to try, in addition to the default algorithm.

Trivial to run the test, it just burns computer cycles... And one never knows when something unusual will come out of such a test, I have been surprised in the past more than once where something I thought would work did not, and something I thought would not work actually did.

I suggest 5 minutes per game+5 seconds per move time control
depths limit to try 16 and 22

I already have a 5+3 match running, the values I chose were 15, 20 and "normal". Will probably be tomorrow until the entire mess finishes...

[Uri Blass]

If absolutely nobody but you can see the point in a particular test (such as those where you want to measure the effect of too low values of MAX_PLY... I don't see anyone protesting Marco's exclamation that those are "madness"), then maybe you should just consider the possibility that the others are right and that you might be wrong. And if you then still cannot see the problem, maybe just accept anyway that it is better to run such tests privately and not irritate everybody else.

I have nothing against your good ideas, but madness is madness.

I think the idea has merit and see the point in this test. I assume Dr. Hyatt does also, since he's devoting his cluster resources to trying it. Others have also expressed interest. Uri's not utilizing the stockfish test system resources for this idea so what's your problem? Your claim of 'madness' is also based on no data.

Note that I don't think it will work, but I am willing to run tests to measure the effect. The two things I need are (a) a suitable time control that is not nuts (40 moves in 2 hours is a bit much for 30K games), and (b) one or two depth limits to try, in addition to the default algorithm.

Trivial to run the test, it just burns computer cycles... And one never knows when something unusual will come out of such a test, I have been surprised in the past more than once where something I thought would work did not, and something I thought would not work actually did.

I suggest 5 minutes per game+5 seconds per move time control
depths limit to try 16 and 22

I already have a 5+3 match running, the values I chose were 15, 20 and "normal". Will probably be tomorrow until the entire mess finishes...

Thanks
one question.
What is the speed loss from checking the additional condition for null move pruning(analysis should be the same in the first 20 plies).

I guess that after optimizations it is clearly less than 1% and has no effect on the result but I want to be sure that I am right about it.

[bob]

If absolutely nobody but you can see the point in a particular test (such as those where you want to measure the effect of too low values of MAX_PLY... I don't see anyone protesting Marco's exclamation that those are "madness"), then maybe you should just consider the possibility that the others are right and that you might be wrong. And if you then still cannot see the problem, maybe just accept anyway that it is better to run such tests privately and not irritate everybody else.

I have nothing against your good ideas, but madness is madness.

I think the idea has merit and see the point in this test. I assume Dr. Hyatt does also, since he's devoting his cluster resources to trying it. Others have also expressed interest. Uri's not utilizing the stockfish test system resources for this idea so what's your problem? Your claim of 'madness' is also based on no data.

Note that I don't think it will work, but I am willing to run tests to measure the effect. The two things I need are (a) a suitable time control that is not nuts (40 moves in 2 hours is a bit much for 30K games), and (b) one or two depth limits to try, in addition to the default algorithm.

Trivial to run the test, it just burns computer cycles... And one never knows when something unusual will come out of such a test, I have been surprised in the past more than once where something I thought would work did not, and something I thought would not work actually did.

I suggest 5 minutes per game+5 seconds per move time control
depths limit to try 16 and 22

I already have a 5+3 match running, the values I chose were 15, 20 and "normal". Will probably be tomorrow until the entire mess finishes...

Thanks
one question.
What is the speed loss from checking the additional condition for null move pruning(analysis should be the same in the first 20 plies).

I guess that after optimizations it is clearly less than 1% and has no effect on the result but I want to be sure that I am right about it.

I could not measure any difference in speed.

[bob]

If absolutely nobody but you can see the point in a particular test (such as those where you want to measure the effect of too low values of MAX_PLY... I don't see anyone protesting Marco's exclamation that those are "madness"), then maybe you should just consider the possibility that the others are right and that you might be wrong. And if you then still cannot see the problem, maybe just accept anyway that it is better to run such tests privately and not irritate everybody else.

I have nothing against your good ideas, but madness is madness.

I think the idea has merit and see the point in this test. I assume Dr. Hyatt does also, since he's devoting his cluster resources to trying it. Others have also expressed interest. Uri's not utilizing the stockfish test system resources for this idea so what's your problem? Your claim of 'madness' is also based on no data.

Note that I don't think it will work, but I am willing to run tests to measure the effect. The two things I need are (a) a suitable time control that is not nuts (40 moves in 2 hours is a bit much for 30K games), and (b) one or two depth limits to try, in addition to the default algorithm.

Trivial to run the test, it just burns computer cycles... And one never knows when something unusual will come out of such a test, I have been surprised in the past more than once where something I thought would work did not, and something I thought would not work actually did.

I suggest 5 minutes per game+5 seconds per move time control
depths limit to try 16 and 22

I already have a 5+3 match running, the values I chose were 15, 20 and "normal". Will probably be tomorrow until the entire mess finishes...

Thanks
one question.
What is the speed loss from checking the additional condition for null move pruning(analysis should be the same in the first 20 plies).

I guess that after optimizations it is clearly less than 1% and has no effect on the result but I want to be sure that I am right about it.

I could not measure any difference in speed.

the tests for 15 and 20 are done. 15 is a -7 Elo change over using 20. The test is "do null if depth < n, where n is 15 or 20. Cluster is not idle, so I am competing with someone else for nodes, which is stretching the 5+5 (I thought it was 5+3 but it is 5+5 time control) game matches out longer than normal.

[bob]

Getting close to done.

Default is 5 better than using the depth<20 to trigger null-move, and 12 better than using depth<15.

Will post final numbers when test is finished. Still lots of competition for cluster nodes during the day.

[bob]

Getting close to done.

Default is 5 better than using the depth<20 to trigger null-move, and 12 better than using depth<15.

Will post final numbers when test is finished. Still lots of competition for cluster nodes during the day.

Had forgotten to check. Here's the final results:

Code: Select all

    Crafty-24.0-1        2641    3    3 30080   58%  2578   32%
    Crafty-24.0R05-20    2638    3    3 30080   58%  2578   32%
    Crafty-24.0R05-15    2632    3    3 30080   57%  2578   33%

24.0-1 is normal Crafty,
24.0-R05-20 uses the depth < 20 limit to do a null-move search
24.0-R05-15 uses the depth < 15 limit

Not much different between default and 20, but didn't really expect much since 20 plies in blitz is limited compared to 15 plies or less.

[zullil]

Getting close to done.

Default is 5 better than using the depth<20 to trigger null-move, and 12 better than using depth<15.

Will post final numbers when test is finished. Still lots of competition for cluster nodes during the day.

Had forgotten to check. Here's the final results:

Code: Select all

    Crafty-24.0-1        2641    3    3 30080   58%  2578   32%
    Crafty-24.0R05-20    2638    3    3 30080   58%  2578   32%
    Crafty-24.0R05-15    2632    3    3 30080   57%  2578   33%

24.0-1 is normal Crafty,
24.0-R05-20 uses the depth < 20 limit to do a null-move search
24.0-R05-15 uses the depth < 15 limit

Not much different between default and 20, but didn't really expect much since 20 plies in blitz is limited compared to 15 plies or less.

I'm just a casual observer, but thanks for doing this.

So your suspicion here is that, had the time control allowed more searches to exceed depth 20, then Crafty-24.0R05-20 would have fared even worse against Crafty 24.0-1?

[Uri Blass]

Getting close to done.

Default is 5 better than using the depth<20 to trigger null-move, and 12 better than using depth<15.

Will post final numbers when test is finished. Still lots of competition for cluster nodes during the day.

Had forgotten to check. Here's the final results:

Code: Select all

    Crafty-24.0-1        2641    3    3 30080   58%  2578   32%
    Crafty-24.0R05-20    2638    3    3 30080   58%  2578   32%
    Crafty-24.0R05-15    2632    3    3 30080   57%  2578   33%

24.0-1 is normal Crafty,
24.0-R05-20 uses the depth < 20 limit to do a null-move search
24.0-R05-15 uses the depth < 15 limit

Not much different between default and 20, but didn't really expect much since 20 plies in blitz is limited compared to 15 plies or less.

I'm just a casual observer, but thanks for doing this.

So your suspicion here is that, had the time control allowed more searches to exceed depth 20, then Crafty-24.0R05-20 would have fared even worse against Crafty 24.0-1?

It is obvious that in theory with infinite time control the version without null move pruning is going to be better because both are going to search to the end of the game but the version without null move pruning are not going to be blind to zugzwang.

In other words Crafty-24.0R05-20 is going to fare better.

The only way to know if Crafty24.0R05-20 is going to fare better or worse in slower practical time control like 20+20 instead of 5+5 is by testing.

I have no strong opinion about it.

[syzygy]

Getting close to done.

Default is 5 better than using the depth<20 to trigger null-move, and 12 better than using depth<15.

Will post final numbers when test is finished. Still lots of competition for cluster nodes during the day.

Had forgotten to check. Here's the final results:

Code: Select all

    Crafty-24.0-1        2641    3    3 30080   58%  2578   32%
    Crafty-24.0R05-20    2638    3    3 30080   58%  2578   32%
    Crafty-24.0R05-15    2632    3    3 30080   57%  2578   33%

24.0-1 is normal Crafty,
24.0-R05-20 uses the depth < 20 limit to do a null-move search
24.0-R05-15 uses the depth < 15 limit

Not much different between default and 20, but didn't really expect much since 20 plies in blitz is limited compared to 15 plies or less.

I'm just a casual observer, but thanks for doing this.

So your suspicion here is that, had the time control allowed more searches to exceed depth 20, then Crafty-24.0R05-20 would have fared even worse against Crafty 24.0-1?

It is obvious that in theory with infinite time control the version without null move pruning is going to be better because both are going to search to the end of the game but the version without null move pruning are not going to be blind to zugzwang.

In other words Crafty-24.0R05-20 is going to fare better.

With INFINITE TIME, yes.

This discussion is going nowhere...

[bob]

Getting close to done.

Default is 5 better than using the depth<20 to trigger null-move, and 12 better than using depth<15.

Will post final numbers when test is finished. Still lots of competition for cluster nodes during the day.

Had forgotten to check. Here's the final results:

Code: Select all

    Crafty-24.0-1        2641    3    3 30080   58%  2578   32%
    Crafty-24.0R05-20    2638    3    3 30080   58%  2578   32%
    Crafty-24.0R05-15    2632    3    3 30080   57%  2578   33%

24.0-1 is normal Crafty,
24.0-R05-20 uses the depth < 20 limit to do a null-move search
24.0-R05-15 uses the depth < 15 limit

Not much different between default and 20, but didn't really expect much since 20 plies in blitz is limited compared to 15 plies or less.

I'm just a casual observer, but thanks for doing this.

So your suspicion here is that, had the time control allowed more searches to exceed depth 20, then Crafty-24.0R05-20 would have fared even worse against Crafty 24.0-1?

Yes. I don't know how much worse. But this is one of those ideas that cluster testing has problems with, because when you screw around with the search, things change at longer time controls and you would have no idea this idea had hurt if I had used 10s + 0.1s for the time control...