Dann Corbit wrote:CThinker wrote:bob wrote:CThinker wrote:Even in crafty, not all nodes are equal. Some nodes are more expensive than others. NPS for crafty is actually 'average' NPS. A cheap (quick) qsearch node is one where eval hit a 'lazy' score, and returned quickly. An expensive (slow) qsearch node is one where eval had to do every line of code in the eval function.
Crafty also does not count nodes each time it calls the MakeMove() function, even though one would claim that it is the the definition of a node. In crafty, there are more calls to the MakeMove() function than there are node counts. This is because crafty calls MakeMove() even for invalid moves. In crafty, node count is done at the start of search/qsearch.
I agree with Vas in that I think that an engine's statistics is specific to the engine. It only makes sense to that engine. As long it is possible to interpret the statistics and that they are consistent, then that I think is good enough.
I myself don't use NPS as a gauge. Instead, I run the profiler and look at its output. Then I can tell how much CPU time is spent on each function and how many times each function is called. I do that during code development. When the engine is released and in use, I no longer care what the NPS is - I already know during development time how the engine behaves. But that is just me...
The issue is nodes however. And whether you count illegal moves or not is not a very big deal. There's a huge difference between whether you choose to count nodes as legal moves only, or as every move you make/unmake, as opposed to making up numbers that are absolutely meaningless intentionally...
The first two cases will be very close overall... if you make up numbers nobody has a clue about what the search is doing...
I think users don't have to know what the search is doing. Whatever it is doing is specific to that engine. I don't see the value in having to expose the details of engine implementation to end users.
I am also not convinced that Vas is just making up numbers. I believe that whatever it is that he is counting is not the same as what others would call nodes. And I think that is what Vas is guilty of - a mistake in terminology. Instead of calling his nodes, "nodes", he should call it "Rybka nodes".
I have here an example on the other direction. Let's say that everyone agrees that a node count is defined as a call to MakeMove(). Now, I implement an engine this way:
1. Increment node count at the start of MakeMove().
2. Always generate 'all' moves in Search (like Rebel does, to have better move ordering).
3. Always use SEE to order all moves (even non-captures, to determine if a piece in a non-capture piece is being given away for free; again, as hinted by Ed in Rebel).
4. Use MakeMove()/UnmakeMove in SEE to do the capture sequence. This avoids the problem with pinned pieces that plague traditional SEE.
For that engine, the node count will be outrageously large. People would complain that I am making up the numbers.
The really big numbers still make sense to me as the developer. I know what the engine is doing.
As for Vas and Rybka, my guess is that the counter is at the start of Search(), and there is no counter in QSearch() or anywhere else. In Thinker, the ratio between calls to Search() and calls to QSearch() is something like 1:200. This looks more like the kind of low numbers that Rybka displays.
So, again, I don't think there is intentional misleading here. Just a difference is terminology and what is being counted.
Junior counts nodes differently. For instance, single reply extensions are not counted as a full node.
I think that if there were a clear definition of what a node is, then a node count would be meaningful.
Do we want to say that every entrance into alpha-beta is a node? If it is, what happens when we circumvent the search with a hash lookup. Is it a node or not?
If a search circumvents quiesce() is it still a node?
We could also count eval() calls, but eval() can also be hashed.
Hashing is irrelevant in both cases. If you count calls to search/quiesce, then you are really counting the classic definition of a node. Once you get into search you might get a hash hit and return, but that was still a node that just got a quick cutoff. Ditto for eval caches. That just saves work at a node, it does not eliminate the fact that a node was reached.
One program does a full minimax, has no hash table of any kind, and so it produces a node number we can understand.
Another program has all sorts of reductions like alpha-beta, null move, lmr, etc, and does not consider single reply extensions as nodes.
I don't know where that comes from. Junior _does_ count nodes just like we do. It doesn't count _plies_ the same way, which is why it reports big search depths. Junior simply adjusts the remaining depth by subtracting 1 or 2, where 1 is for captures, 2 is for non-captures, and maybe even 0 for checks. So rather than extending, it just subtracts less which is the same thing. But the node counts are very comparable between junior and another program, but not the search depths reported.
If we compare the numbers from these two programs what will we learn?
If we do not know the kind of reductions that are going on inside of a program, I submit that node counts are almost completely meaningless.
I submit that if you report a valid node count and an understandable search depth, I can figure out what you are doing inside your search with respect to extensions and reductions. And that is the point of obfuscating the numbers, to make this more difficult.