Interpretation of a result for time and node count

[bob]

Interpretation of a result for time and node count:

When reporting the result of a search, many programs (maybe nearly all of them) report the total time used and total node count. But are these an accurate description of the effort needed to produce the result?

I suggest that a better report would use the time/nodes numbers at the point in the search when the result PV and score were established. Alternatively, the time/nodes numbers reported would be from the point in the search after the point at which any PV could be established; i.e., the last time a score/bound was determined for a root move selection. The idea is to not credit any effort in which all data generated was thrown away when a resource limit (e.g., allocated elapsed time) was encountered.

The change is easy to implement. A program need only capture the time/nodes numbers at the time a PV is established and pass these along with the final PV when the result report is generated.

I'm not sure I follow. Don't you need to search _every_ root move before you know what the final result (best move) is???

[sje]

I suggest that a better report would use the time/nodes numbers at the point in the search when the result PV and score were established. Alternatively, the time/nodes numbers reported would be from the point in the search after the point at which any PV could be established; i.e., the last time a score/bound was determined for a root move selection. The idea is to not credit any effort in which all data generated was thrown away when a resource limit (e.g., allocated elapsed time) was encountered.

I'm not sure I follow. Don't you need to search _every_ root move before you know what the final result (best move) is???

Consider a traditional program that's halfway through the nth iteration. It has a PV set by one of the root moves already considered at the iteration, but decides that there's not enough time to fully search the remaining root moves. So there's a final result although not all the root moves have been searched, at least at that iteration.

In extremely tight time pressure, perhaps not all of the moves in even the first iteration can be searched fully, but there still may be a PV. Of course, not all programs may behave this way.

[sje]

Reporting the wall clock elapsed time is simple and easily understood, but it carries the least meaning. Also, it's redundant assuming a GUI or a server is in use, as either can easily calculate the elapsed time.

Not at all. For one, normally (i.e. when not in nps mode) the GUI flags the engine based on wall-clock time, so the engine has to base its timing decisions on that, even if it only gets a small share of it in terms of CPU usage. The time measured by the GUI will not be the same as the time used by the engine because of communication delays, and being able to see the difference is often a great help in debugging.

Of course a program has to consider wall time. But it doesn't have to report wall time. Indeed, a GUI or server shouldn't even consider a program's wall time report -- the program could be cheating!

Lag time is different subject. A lag time can be sent to a program, but it of course cannot report it's own lag.

[sje]

I suggest that a better report would use the time/nodes numbers at the point in the search when the result PV and score were established.

How much better? By what standard and by what amount? (Why not report Joules consumed, for example.)

Long ago I suggested a per-game joule limit instead of a time control; it would be a great equalizer between a program running on a notebook and one running on eight racks filled with compute nodes.