Interpretation of a result for time and node count

[sje]

Reporting the wall clock elapsed time is simple and easily understood, but it carries the least meaning. Also, it's redundant assuming a GUI or a server is in use, as either can easily calculate the elapsed time.

The most informative reply is one that's the sum of the physical core usages. This, combined with knowledge of the CPU model and clock speeds (as might appear in server finger notes), gives the best estimate of the total computing effort.

[sje]

Reporting a node count that includes illegal positions has a long heritage. The practice goes back to the Bronze Age at least, including Jim Gillogly's program Tech from 1970.

But reporting illegal positions as nodes is misleading. What analysis can be done with an illegal position? None, except determining that it's illegal and shouldn't have been generated. So it's a waste.

And some programs have legal-only move generators and so will not produce illegal positions. So a comparison of node counts is useless unless the programs involved either both ignore or never generate illegal positions.

[Sven]

Reporting the wall clock elapsed time is simple and easily understood, but it carries the least meaning. Also, it's redundant assuming a GUI or a server is in use, as either can easily calculate the elapsed time.

The most informative reply is one that's the sum of the physical core usages. This, combined with knowledge of the CPU model and clock speeds (as might appear in server finger notes), gives the best estimate of the total computing effort.

But the total computing effort is not the only information you are interested in. Users may be interested in something else: What does this engine find in X seconds elapsed on the wall clock? Play a game with Y seconds increment per move?

You are right when stating that wall clock time is not perfectly accurate compared to CPU time. So there are different requirements to be fulfilled:
- report accurate times based on elapsed CPU time,
- report wall clock times to satisfy user requirements.

You are free to additionally report everything you want, as long as the information required by current protocols (WB/UCI) is also present in the expected format.

Sven

[Sven]

Reporting a node count that includes illegal positions has a long heritage. The practice goes back to the Bronze Age at least, including Jim Gillogly's program Tech from 1970.

But reporting illegal positions as nodes is misleading. What analysis can be done with an illegal position? None, except determining that it's illegal and shouldn't have been generated. So it's a waste.

And some programs have legal-only move generators and so will not produce illegal positions. So a comparison of node counts is useless unless the programs involved either both ignore or never generate illegal positions.

I fully agree that illegal positions should not be counted in principle. Nevertheless, their real influence to the total node count is very marginal in most cases. Comparison is still possible, with a small and acceptable error.

For some engines it is faster not to care about legality during move generation but to postpone legality check until moves are really tried on the board. So your "shouldn't have been generated" does not always apply, it depends on implementation issues. Of course the counting method could take into account legality somehow but it might make things a tiny bit more complex - again, depending on the implementation.

Sven

[sje]

I fully agree that illegal positions should not be counted in principle. Nevertheless, their real influence to the total node count is very marginal in most cases. Comparison is still possible, with a small and acceptable error.

I disagree; the influence can be significant and here's why:

Consider:
[d]3rr1k1/pp3pp1/4b3/8/2P1B2R/6QP/P3q1P1/5R1K w - - 0 1
Neither king is out in the middle of the board, yet there are many check evasion nodes seen in a brief search:

Code: Select all

  Total node counts:
    All: 2,098,996   Capture: 1,367,832   Evasion: 315,285   General: 195,754   PlyZero: 6

For every evasion node, perhaps some ten illegal positions might be considered by some programs. That would be an additional three million plus nodes and that would definitely make a big impact on the total node count.

[sje]

By the way, the aforementioned position is #82 from Reinfeld's Win At Chess.

The nascent CIL Toolkit solved it and gave the first seven moves of the correct PV with a search of 112,963 nodes.

Berliner's program CAPS (or is it "Caps"?) got it with 489 nodes.

Wilkins' program Paradise needed only 36 nodes.

[Sven]

I fully agree that illegal positions should not be counted in principle. Nevertheless, their real influence to the total node count is very marginal in most cases. Comparison is still possible, with a small and acceptable error.

I disagree; the influence can be significant and here's why:

Consider:
[d]3rr1k1/pp3pp1/4b3/8/2P1B2R/6QP/P3q1P1/5R1K w - - 0 1
Neither king is out in the middle of the board, yet there are many check evasion nodes seen in a brief search:

Code: Select all

  Total node counts:
    All: 2,098,996   Capture: 1,367,832   Evasion: 315,285   General: 195,754   PlyZero: 6

For every evasion node, perhaps some ten illegal positions might be considered by some programs. That would be an additional three million plus nodes and that would definitely make a big impact on the total node count.

My old engine Surprise which still counts illegal positions (KnockOut does not) needs 107251 nodes until displaying the solution of that position for the first time. At the end of the corresponding iteration it has searched 150977 nodes of which 7982 are illegal (about 5%). After searching about half a minute Surprise reports 35008511 nodes with 1546457 illegal (about 4,5%). This seems to be a lot but I would still consider 5% as "reasonable error".

(How do you get your 15% illegal positions?) EDIT: I misread, you got 15% *evasion* nodes which is even more surprising for me. But your assumption about the number of illegal nodes per evasion node is not correct, due to alpha/beta cutting off most of them.


Maybe you are right that engines which are not implementing a check evasion generator may have more variance in their node counts.

I am not really defending the "counting illegal positions does not matter at all" side ... I just think it is not a general big problem.

Sven

[hgm]

Reporting the wall clock elapsed time is simple and easily understood, but it carries the least meaning. Also, it's redundant assuming a GUI or a server is in use, as either can easily calculate the elapsed time.

Not at all. For one, normally (i.e. when not in nps mode) the GUI flags the engine based on wall-clock time, so the engine has to base its timing decisions on that, even if it only gets a small share of it in terms of CPU usage. The time measured by the GUI will not be the same as the time used by the engine because of communication delays, and being able to see the difference is often a great help in debugging.

[hgm]

I fully agree that illegal positions should not be counted in principle.

I don't even agree to that. It all depends on how much work you do in an illegal position. If that is just as much work as in any other position, it is totally natural to count it as a node.

In micro-Max I count illegal positions as a node. It still has to generate moves there before it discovers it can capture the King. This is in fact much more work than a node where it stands pat. But I guess futility pruning takes care that I don't have too many of those.

[marcelk]

I suggest that a better report would use the time/nodes numbers at the point in the search when the result PV and score were established.

How much better? By what standard and by what amount? (Why not report Joules consumed, for example.)