What is the Ideal Output for Understanding a Chess Engine?

[rfadden]

In another thread we were discussing the various ways of tallying a Node Count. Here I would like to raise this topic as a separate question.

I realize this topic may come up repeatedly but I have not been reading CCC for a long time and I am hoping that people find this topic worth chatting about here.

So there was mention of counting how many times one enters the recursive search routines in a chess program, and there was some mention of not including Qu search and also of course the statement that Qu search counts must be included.

I am also wondering: if we tend to only look at one number then why not simply look at the number of calls to Eval during search?

I'd really like to hear the thoughts of various people here. I'd like to hear the thoughts of experienced folks on this topic of tallying calls to Eval.
I'm further wondering if we could take these reasons and turn them into an idea something like: Tally two entities, one is calls to eval and the other is a tally of calls to the recursive search functions. All kinds of thoughts pop up like possible interest in comparing these two numbers.

Wouldn't it be typical for someone to reply here that "This has all been discussed previously, and just about everyone agreed the answer is X." Well if someone will say that, the I would like to know what everyone has agreed to. I want to know about "X" the agreed upon technique for Node Count or the concensus on what this should be.

If people are not agreeing on X, then I think the general discussion on figuring out the best technique for X would be educational.

Thanks,
Rick

[Bill Rogers]

One possible problem with you idea is that during a 10 ply search 10 nodes are generated before on evaluation is done. The Eval is done at the top of the search.

[Gerd Isenberg]

Usually a node is entered with a call of a recursive search/qsearch-routine - or equivalently a jump to an entry point inside an iterative search. Cycles per node is a fine measure, or taking energy consumption of certain processor systems into account, nano-joule per node and cycle

For an inner eye following proposal: absolute numbers and some ratios, differentiated by ply (let say 0==root ...50) and thread/process of the search, all numbers distinct after each search iteration as well as aggregated for effective branching factors:

Code: Select all

nodes, 
pv-nodes, 
zerowindow-nodes, 
re-searches, 
qnodes, 
horizon-nodes, 
frontier-nodes, 
pre-frontier-nodes, 
pre-pre-frontier-nodes, 
leaf-nodes, 
nullmove-cuts, 
cut-nodes,
all-nodes, 
cuts-onMove[moveNr 1..N], 
cuts-onHashMove, 
cuts-onKiller[killerIndex],
expected-cut-nodes, 
expected-cut-nodes-becomes-all, 
expected-all-nodes, 
expected-all-nodes-becomes-cut, 
fail-high-on-hashprobe,
fail-low-on-hashprobe,
exacthit-on-hashprobe,
nodes-below-nullmove,
nodes-in-IID,

not to mention the number of various (fractional) extensions, reductions or prunings, mates, stalemates, repetitions, recognizer- and egtb-hits etc.. Calls to lazy-eval versus eval. Number of stores, probes and hits and sufficient hits of various caches since eval may be cached in main transposition table as well as in dedicated eval hash.

Probably more interesting for the programmer than for the average user, anyway we only need to log it, and to visualize it with a neat configurable online chart application for the inner eye. Chess-System Tal was quite nice with that respect. Unfortunately updating all those counters will take some time...

Cheers,
Gerd

[Dann Corbit]

The ideal way to understand a chess engine is to look at the moves that it makes.

[jwes]

f5

[tmokonen]

f5

Is this the chessic equivalent of the number 42?

Tony

[Ovyron]

f5

Is this the chessic equivalent of the number 42?

Tony

As in:

"42 is the answer but nobody knows the question"

"f5 is the best move, but nobody knows the position"

[tmokonen]

f5

Is this the chessic equivalent of the number 42?

Tony

As in:

"42 is the answer but nobody knows the question"

"f5 is the best move, but nobody knows the position"

I think you've just figured out the answer to Life, the Universe, Chess and Everything

[bob]

In another thread we were discussing the various ways of tallying a Node Count. Here I would like to raise this topic as a separate question.

I realize this topic may come up repeatedly but I have not been reading CCC for a long time and I am hoping that people find this topic worth chatting about here.

So there was mention of counting how many times one enters the recursive search routines in a chess program, and there was some mention of not including Qu search and also of course the statement that Qu search counts must be included.

I am also wondering: if we tend to only look at one number then why not simply look at the number of calls to Eval during search?

I'd really like to hear the thoughts of various people here. I'd like to hear the thoughts of experienced folks on this topic of tallying calls to Eval.
I'm further wondering if we could take these reasons and turn them into an idea something like: Tally two entities, one is calls to eval and the other is a tally of calls to the recursive search functions. All kinds of thoughts pop up like possible interest in comparing these two numbers.

Wouldn't it be typical for someone to reply here that "This has all been discussed previously, and just about everyone agreed the answer is X." Well if someone will say that, the I would like to know what everyone has agreed to. I want to know about "X" the agreed upon technique for Node Count or the concensus on what this should be.

If people are not agreeing on X, then I think the general discussion on figuring out the best technique for X would be educational.

Thanks,
Rick

Calls to Evaluate() will almost exactly match the number of calls to Quiesce(), since Evaluate() is called at the top of Quiesce() to set the "stand-pat" score before searching any moves there...

My only comment is that nodes ought to be reported consistently, and in the literature, a "new position" is a new node. You could count calls to MakeMove() since that leads to a new position. You could count calls to Search() / Quiesce() which ought to be the same number. Anything else (Evaluate() calls as an example) will be an example of a number that is less useful since it omits what might be large parts of the tree. For example, chessmaster is very selective, which means not counting the normal search nodes would greatly reduce the total nodes searched number, making it even harder to compere nodes between programs.

I would always suggest sticking with the traditional definition of "node" unless your name happens to be Donald Knuth, Claude Shannon, etc... Then one might have the "reputation" necessary to re-define an accepted term.

If a definition becomes vague, it becomes useless.

[sje]

For its traditional A/B search, Symbolic has a simple convention for node counting: the counter is incremented for each time a legal move is made on the analysis board. It is guaranteed that some sort of score will result from this although it may not result in a full evaluation.

The only non legal moves tried are those pulled from a table that were legal in some other position, but not in the current one. These are not counted. Nearly all moves are legal as the move generators produce only legal moves.

For its cognitive search, Symbolic keeps the whole tree in memory and so the node count is simply the total number of nodes in the tree graph. A purist might consider this an undercount as a given node may be visited more than once for different reasons.