Ponder=On vs. Ponder=Off

[Peter Berger]

These days every ordinary computer or notebook has more than one CPU, mine has two e.g.
I read on this board that given you have to decide between providing more CPUs to engines you pit against each other on a single computer for testing purposes with ponder off and less CPUs with ponder on you'd rather go with the ponder off approach.
I assume that ELO-wise the difference between these approaches is actually negligible ( though I haven't checked this really carefully yet).
My personal impression from watching games for my personal entertainment ( based on way too few games to come up with sth really statistically significant and measurable) is that games tend to be way more interesting and inspiring in general with ponder=on though.
I can provide a clumsy explanation, that makes some sense to me. In case one engine comes up with some viable idea in a game the other one doesn't get at all or misjudges it can gain a lot of depth ( and time on the clock) for a few moves. One of the engines can create a local maximum to enlarge this edge in a given situation this way.
Have others observed sth. similar ( or sth. completely different ) ?
Peter

[bob]

These days every ordinary computer or notebook has more than one CPU, mine has two e.g.
I read on this board that given you have to decide between providing more CPUs to engines you pit against each other on a single computer for testing purposes with ponder off and less CPUs with ponder on you'd rather go with the ponder off approach.
I assume that ELO-wise the difference between these approaches is actually negligible ( though I haven't checked this really carefully yet).
My personal impression from watching games for my personal entertainment ( based on way too few games to come up with sth really statistically significant and measurable) is that games tend to be way more interesting and inspiring in general with ponder=on though.
I can provide a clumsy explanation, that makes some sense to me. In case one engine comes up with some viable idea in a game the other one doesn't get at all or misjudges it can gain a lot of depth ( and time on the clock) for a few moves. One of the engines can create a local maximum to enlarge this edge in a given situation this way.
Have others observed sth. similar ( or sth. completely different ) ?
Peter

It is actually not so negligible.

Take a two core box. You run A vs B, no ponder, both using 2 cores. They don't interfere with each other, both get 200% of normal compute time per move. Pretty good win.

Now run A vs B, both pondering, both using 1 core. When it is A's move, it burns 100% of a CPU, but then ponders while B is thinking. If A predicts correctly 50% of the time, then it gets 1.5X the normal search time on average, 200% when pondering correctly, 100% when the pondered move was wrong.

200% is > 150%. I'd lean toward two threads ponder=off.

As far as your observation goes about the infamous "ponder hit cycle" it is a well-known issue. Annoying as all hell at times, when the game is almost totally forced, and one side makes one unexpected move to break the opponent's pondering. Now the opponent takes the full time while your program ponders for the same time and you make instant move after instant move until your opponent makes a move you don't expect and this flips to his favor for a while.

But that is purely random over enough games. I do test a lot with ponder=on since that behaves differently (time usage and such). And since that is the way we play in real games, you had better test some in that type of game to be sure there are no unexpected program issues such as poor time targets and such.

[kranium]

Now run A vs B, both pondering, both using 1 core. When it is A's move, it burns 100% of a CPU, but then ponders while B is thinking. If A predicts correctly 50% of the time, then it gets 1.5X the normal search time on average, 200% when pondering correctly, 100% when the pondered move was wrong.

Agreed...pondering can pretty inefficient. Perhaps it could/would be better if implemented differently.
For ex: why not simply analyze the current position instead of being forced to guess what the position might be after your opponents turn?

[Robert Pope]

Agreed...pondering can pretty inefficient. Perhaps it could/would be better if implemented differently.
For ex: why not simply analyze the current position instead of being forced to guess what the position might be after your opponents turn?

I think ponder matches are around 80%. So if you search the current position instead of the ponder position, you get 1 less ply of searching 80% of the time.

[kranium]

Hi Robert-

The only data I know of was compiled by CCRL here:
http://computerchess.org.uk/ccrl/4040/c ... ilar+pairs
http://computerchess.org.uk/ccrl/4040/c ... rent+pairs

The most similar pairs (of engines) get around 70-78% ponderhit
The most different pairs (of engines) get around 14-18% ponderhit
so, perhaps 50% might be a good guess at the average # of ponderhits for all engines

This means that during a match, when pondering, the engine will waste 50% of that time...it's analyzing positions that cannot occur (excluding thru possible transpositions).
The hash table will fill (mainly) with completely unusable info.

But, if the engine was busy analyzing the 'current' position instead the resulting position after it's wrongly 'guessed' ponder move, the hash table would be full of relevent and useful data...
losing 1 ply search depth might be (seems) a small price to pay...

It would interesting to see the results of testing on this.

Regards,
Norm

[bob]

Now run A vs B, both pondering, both using 1 core. When it is A's move, it burns 100% of a CPU, but then ponders while B is thinking. If A predicts correctly 50% of the time, then it gets 1.5X the normal search time on average, 200% when pondering correctly, 100% when the pondered move was wrong.

Agreed...pondering can pretty inefficient. Perhaps it could/would be better if implemented differently.
For ex: why not simply analyze the current position instead of being forced to guess what the position might be after your opponents turn?

Simple answer. You spend too little time on the right move for your opponent. So you can't move instantly once he moves. With normal pondering, 1/2 of the time you can move instantly. Looking at the current position, you will never move instantly. And given the fact that you are searching every possible opponent move, you will have expended only minimal effort on the move he actually plays...

[kranium]

Now run A vs B, both pondering, both using 1 core. When it is A's move, it burns 100% of a CPU, but then ponders while B is thinking. If A predicts correctly 50% of the time, then it gets 1.5X the normal search time on average, 200% when pondering correctly, 100% when the pondered move was wrong.

Agreed...pondering can pretty inefficient. Perhaps it could/would be better if implemented differently.
For ex: why not simply analyze the current position instead of being forced to guess what the position might be after your opponents turn?

Simple answer. You spend too little time on the right move for your opponent. So you can't move instantly once he moves. With normal pondering, 1/2 of the time you can move instantly. Looking at the current position, you will never move instantly. And given the fact that you are searching every possible opponent move, you will have expended only minimal effort on the move he actually plays...

yes, but then of course, if ponderhit = 'no', instead of making an instant move, you spend the 'normal' amount of time searching as always...just as if ponder = off

and don't forget for some engines, ponder hit is as low as 10-20%, and many engines don't make 'instant moves' at all

the question is: which scheme might yield best results...
I don't think any data has been collected on this and I'm a bit doubtful the current UCI an WB impementation is best

Are you one of the originators? (has it ever been tested?)

[hgm]

I don't think this has been sufficiently tested, and the ponder-hit statistics is misleading. There is always a fair number of 'easy moves' in a game, where there basically all alternatives are immediately losing. (E.g. recaptures.) Of course these will give you ponder hits. But usually the opponent also moves quickly on them, so the ponder hit gains you very little thinking time.

Note that in-between implementations are also possible. Shokidoki first ponders the position until a certain depth is reached (which can be reached immediately through a hash hit in the root), and then freezes the best move, so that the remaining ponder time is spent on that move only. This also solves the problem in situations where you do not have a ponder move (e.g. because the previous move was a 'single legal move' situation where you did not search at all, but just played the move).

[bob]

Now run A vs B, both pondering, both using 1 core. When it is A's move, it burns 100% of a CPU, but then ponders while B is thinking. If A predicts correctly 50% of the time, then it gets 1.5X the normal search time on average, 200% when pondering correctly, 100% when the pondered move was wrong.

Agreed...pondering can pretty inefficient. Perhaps it could/would be better if implemented differently.
For ex: why not simply analyze the current position instead of being forced to guess what the position might be after your opponents turn?

Simple answer. You spend too little time on the right move for your opponent. So you can't move instantly once he moves. With normal pondering, 1/2 of the time you can move instantly. Looking at the current position, you will never move instantly. And given the fact that you are searching every possible opponent move, you will have expended only minimal effort on the move he actually plays...

yes, but then of course, if ponderhit = 'no', instead of making an instant move, you spend the 'normal' amount of time searching as always...just as if ponder = off

and don't forget for some engines, ponder hit is as low as 10-20%, and many engines don't make 'instant moves' at all

the question is: which scheme might yield best results...
I don't think any data has been collected on this and I'm a bit doubtful the current UCI an WB impementation is best

Are you one of the originators? (has it ever been tested?)

I have certainly tested various approaches:

(1) ponder predicted move (what I do today)

(2) ponder predicted move until it fails high significantly, then try pondering another move

(3) ponder all moves (search for other side) which guarantees a ponder hit, but you only spend a fraction of the pondering time on the move played since you also spent time searching moves that were not played.

(4) ponder just a subset of legal moves. Similar to (3) but lets you spend more time on each move compared to (3).

I don't consider ponder hit statistics very revealing. Playing much weaker programs will drop ponder hits significantly, but you still win. It is playing against equal or stronger opponents where this is most productive. And in those games ponder hit rates are usually well above 50%. So long as that is true, it is pretty intuitive that pondering just one move is the right way to do this. Crafty counts "hits" for informational reasons. I just looked at a couple of games played on ICC recently against strong computer opponents. The first game had 35 ponder hits out of 53 total moves played. A couple of others had 40 out of 62, 38 out of 58, 36 out of 75, 31 out of 58, etc. So generally 50% or better, most of the time better. So long as you hit 50%, there's no way to ponder more effectively, because this will save you 50% of the time used on searches or better, and I can't imagine any other approach that can come close. Only when you drop to less than 50% by a significant margin do you have an opportunity to do something different that might be more effective.

[bob]

I don't think this has been sufficiently tested, and the ponder-hit statistics is misleading. There is always a fair number of 'easy moves' in a game, where there basically all alternatives are immediately losing. (E.g. recaptures.) Of course these will give you ponder hits. But usually the opponent also moves quickly on them, so the ponder hit gains you very little thinking time.

Note that in-between implementations are also possible. Shokidoki first ponders the position until a certain depth is reached (which can be reached immediately through a hash hit in the root), and then freezes the best move, so that the remaining ponder time is spent on that move only. This also solves the problem in situations where you do not have a ponder move (e.g. because the previous move was a 'single legal move' situation where you did not search at all, but just played the move).

Crafty is never without a ponder move. It has three options.

(1) ponder move from PV (normal case).

(2) if none, probe hash table and ponder best move returned.

(3) if that fails, switch sides, do what I call a "puzzling" search (puzzling over which move to ponder) for 1/10th the normal target time, and then switch back to the normal side and ponder that move.

The last option sounds similar to what you described. But given a choice, I'll take the PV move any time, since it has had more effort expended searching it (compared to 1/10th the normal time for (3) above)

BTW, Crafty has another option for pondering while still in book. In "tournament mode" I generate the root move list, which is the set of legal moves in this position. I then exclude any of those moves that are found with a book probe. From the remaining set, I again do the 1/10th normal time search to choose the best, then ponder that. I did this back in the days of Cray Blitz even, the idea being that if I play a move that takes my opponent out of book, in many cases his move is reasonably obvious. If his best move is a book move, I will have a reply instantly, already, so I try to find the best non-book move, and then ponder that, so that if he plays a non-book move there's a pretty good chance he will play the move I am pondering and I still move very quickly after he does.