Elo points gain from doubling time

[lkaufman]

I did a similar partial study for Critter 1.2 and 1.6, to see if the pattern is much different.

As far as I know, Critter 1.2 does not have any search feature other than Singular Extension that kicks in at 9 ply, so this would appear to fully account for the observed jump in Critter

Easy to verify. Retry the 8 vs 9 ply match with singular extensions disabled and compare the results.

Good suggestion, thanks, I think I'll try that. Any prediction as to what will happen?

[Uri Blass]

The point of HGM is that it may be better to start engines matches not from balanced positions if balanced positions give more than 90% draws and the better engine may score only 1.04 out of 2.

It may be better to start from unbalanced positions when often white wins and often it is a draw and the better engine may score in most of the cases 1.5 out of 2)

An example for unbalanced position is the position after
1.e4 f6 or the position after 1.e4 a6 or the position after 1.d4 e5

Not sure if the result between very strong engines may be often 1.5 out of 2 for white(maybe white is simply winning after 1.e4 f6 at the very high level but the idea is that you start from a position when white has a big advantage and at the very high level the only question is if white can win or black can save a draw)

[Don]

The point of HGM is that it may be better to start engines matches not from balanced positions if balanced positions give more than 90% draws and the better engine may score only 1.04 out of 2.

It may be better to start from unbalanced positions when often white wins and often it is a draw and the better engine may score in most of the cases 1.5 out of 2)

An example for unbalanced position is the position after
1.e4 f6 or the position after 1.e4 a6 or the position after 1.d4 e5

Not sure if the result between very strong engines may be often 1.5 out of 2 for white(maybe white is simply winning after 1.e4 f6 at the very high level but the idea is that you start from a position when white has a big advantage and at the very high level the only question is if white can win or black can save a draw)

I thought of creating a book for testing that starts from positions generated by random moves. Some of them would end with a clear winning advantage, even a forced checkmate that most programs would see. But such a book would be rather artificial and would not reflect positions that chess programs need to know about. But it would definitely create auto-test scores with many less draws while giving superior programs a real chance to strut their stuff.

[lkaufman]

I did a similar partial study for Critter 1.2 and 1.6, to see if the pattern is much different.

As far as I know, Critter 1.2 does not have any search feature other than Singular Extension that kicks in at 9 ply, so this would appear to fully account for the observed jump in Critter

Easy to verify. Retry the 8 vs 9 ply match with singular extensions disabled and compare the results.

Okay, I did it. This time I got 157.2 elo (after 7k games), a dramatic reduction from 236.9. As a control, I reran 7 vs 8 ply, and got 180.0, up a bit from 173.6. I also reran 10 vs 9 ply (which also benefits from SE), and got 138.9 elo, down sharply from 175.2.

Conclusion: Singular Extension accounts for all the anomalies observed with the Critter matches. Without it the elo gain decays as it should from 180.0 to 157.2 to 138.9 at these depths. So there is no reason to think that this would not also fully explain the spike in Houdini 2 at depth 9, since the behavior was so similar.

[rvida]

I did a similar partial study for Critter 1.2 and 1.6, to see if the pattern is much different.

As far as I know, Critter 1.2 does not have any search feature other than Singular Extension that kicks in at 9 ply, so this would appear to fully account for the observed jump in Critter

Easy to verify. Retry the 8 vs 9 ply match with singular extensions disabled and compare the results.

Okay, I did it. This time I got 157.2 elo (after 7k games), a dramatic reduction from 236.9. As a control, I reran 7 vs 8 ply, and got 180.0, up a bit from 173.6. I also reran 10 vs 9 ply (which also benefits from SE), and got 138.9 elo, down sharply from 175.2.

Conclusion: Singular Extension accounts for all the anomalies observed with the Critter matches. Without it the elo gain decays as it should from 180.0 to 157.2 to 138.9 at these depths. So there is no reason to think that this would not also fully explain the spike in Houdini 2 at depth 9, since the behavior was so similar.

Mystery solved

I wonder whether Stockfish has a similar spike? SF starts doing singular extensions at different depths for PV and non-PV nodes:

Code: Select all

  // Minimum depth for use of singular extension
  const Depth SingularExtensionDepth[] = { 8 * ONE_PLY, 6 * ONE_PLY };

  /* ... */

    singularExtensionNode =   !RootNode
                           && !SpNode
                           && depth >= SingularExtensionDepth[PvNode]
                           && ttMove != MOVE_NONE
                           && !excludedMove // Recursive singular search is not allowed
                           && (tte->type() & VALUE_TYPE_LOWER)
                           && tte->depth() >= depth - 3 * ONE_PLY; 

[hgm]

But such a book would be rather artificial and would not reflect positions that chess programs need to know about.

That is a very dangerous assumption. Truth is that you don't know what positions your program needs to know about. If you want it to be any good for other purposes than playing games, that is. And playing games is only a minority application; most customers will use engines for analysis. And you cannot know which crazy positions the want to analyze.

Once engines would be asymptotically strong, they might only play draws in games. So it would become totally irrelevant to them to know how to win KQK or KPK. Because these positions are not drawn, and would thus never occur in their games. So they would be grouped under "positions that chess programs don't need to know about". Which would not go down very well with the average user...

A good engine needs to know about all positions. Also those that can only be achieved by poor play.

[Don]

But such a book would be rather artificial and would not reflect positions that chess programs need to know about.

That is a very dangerous assumption. Truth is that you don't know what positions your program needs to know about. If you want it to be any good for other purposes than playing games, that is. And playing games is only a minority application; most customers will use engines for analysis. And you cannot know which crazy positions the want to analyze.

Actually, I have posted on that subject many times. I have argued that testing with random positions might make your program more robust in some sense because programs in my opinion do tend to be "tuned" to the opening position and all that it implies.

So I would take some sort of measured approach to this. Athletes always train with a high level of specificity, but they also do supplemental training. It would not make much sense for a weightlifter to do marathon training or a football player to spend most of his time "working out" with his golfing technique, but it does make some sense for sprinters to do some weight lifting, tennis players to do some endurance training, etc ... But of course you still spend most of your focus on the thing you expect to be doing in competition.

Once engines would be asymptotically strong, they might only play draws in games. So it would become totally irrelevant to them to know how to win KQK or KPK. Because these positions are not drawn, and would thus never occur in their games. So they would be grouped under "positions that chess programs don't need to know about". Which would not go down very well with the average user...

A good engine needs to know about all positions. Also those that can only be achieved by poor play.

[hgm]

I heard that in training for the high jump top athletes actually never make any jumps. I guess it depends on how difficult it is to master and perfect your technique, vs building more muscle power. Apparently the high jump is technically easy.

[Don]

I heard that in training for the high jump top athletes actually never make any jumps. I guess it depends on how difficult it is to master and perfect your technique, vs building more muscle power. Apparently the high jump is technically easy.

I find that very difficult to believe. However I can imagine that there are some sports where it's counterproductive to be too specific.

I can think of boxers - I know they do a combination of different kinds of training including sparing with various sparing partners, but I doubt they go in the ring and take a serious beating every day, for example the same beating they would take in a real match.

[lkaufman]

I did a similar partial study for Critter 1.2 and 1.6, to see if the pattern is much different.

As far as I know, Critter 1.2 does not have any search feature other than Singular Extension that kicks in at 9 ply, so this would appear to fully account for the observed jump in Critter

Easy to verify. Retry the 8 vs 9 ply match with singular extensions disabled and compare the results.

Okay, I did it. This time I got 157.2 elo (after 7k games), a dramatic reduction from 236.9. As a control, I reran 7 vs 8 ply, and got 180.0, up a bit from 173.6. I also reran 10 vs 9 ply (which also benefits from SE), and got 138.9 elo, down sharply from 175.2.

Conclusion: Singular Extension accounts for all the anomalies observed with the Critter matches. Without it the elo gain decays as it should from 180.0 to 157.2 to 138.9 at these depths. So there is no reason to think that this would not also fully explain the spike in Houdini 2 at depth 9, since the behavior was so similar.

Mystery solved

I wonder whether Stockfish has a similar spike? SF starts doing singular extensions at different depths for PV and non-PV nodes:

Code: Select all

  // Minimum depth for use of singular extension
  const Depth SingularExtensionDepth[] = { 8 * ONE_PLY, 6 * ONE_PLY };

  /* ... */

    singularExtensionNode =   !RootNode
                           && !SpNode
                           && depth >= SingularExtensionDepth[PvNode]
                           && ttMove != MOVE_NONE
                           && !excludedMove // Recursive singular search is not allowed
                           && (tte->type() & VALUE_TYPE_LOWER)
                           && tte->depth() >= depth - 3 * ONE_PLY; 

I would predict two spikes, at depths 7 and 9. I'll try it later today.