Hamsters randomizer in action

[hgm]

Somehow I would expect behavior like this in Pawn endings: one side wins a Pawn, but only at the expense of giving the opponent free passages to Pawns on the other wing, which it then unavoidably gobbles up a few ply later. So the Pawn gain turned out to be an illusion, and the PV switches to taking opposition to block break through of the enemy King, and abstaining from going for his Pawns. Until a few ply deeper it turns out that going for his Pawn makes a passer for you that promotes first if he tries clearing out your other wing. Etc.

To dampen this out requires a smart eval that recognizes and score promotion races or break-throughs. Or extensions that accomplish the equivalent. Otherwise behavior like this seems quite natural.

Note that things like this can have noting to do with complaints that your engine is "too volatile to test". If its results ove 100 games spread between 25% and 75%, this can only be explained by the individual games not being independent. For independent games the standard deviation of scores from a 100-game match will always be the standard deviation of a single match divided by sqrt(100) = 10. So for the standard deviation of a 100-game match to be 25%, the standard deviation of an individual game should be 2.5 points, i.e. you should score 3 points per games as often as -2 points (or even more extreme if the more 'drawish' scores 0, 1, and 2 occurred as well). It is hard to believe that any search instability in your engine could cause such extreme game results...

[Tony]

Hi Harm Geert,

I'm not convinced yet What if the engine is better in the first part of the game ? Suppose the pawn ending comes up, say 50% of the time. If it shows up you loose, if it doesn't you win. Chances that it shows up 5 in a row, (or doesn't, 5 in a row) are not low.

Couldn't that explain the deviation ?

Tony

[hgm]

The key phrase here is "in a row". If subsequent games are independent, the probability for it to show up should not depend on the fact if it showed up in the previous game.

So at least one of the two probabilities you mention (five times in a row, or five times in a row not) must be smaller than 1:32. They can both be 1:32 if they show up in 50% of the cases in individual games.

This is not a large probability. Of course the probability to occur 5 times in a row would get very large if the probability to occur in an individual game was very close to 1. But then it would not be a source of variability, as it would always occur a lot, and the probability that a number of times in a row it would not occur becomes vanishingly small.

The standard deviation of a single coin flip with outcome 0 or 1 is sqrt(P(0)*P(1)). This is maximum when P(0) = P(1) = 0.5, namely 0.5. Non-zero draw probability reduces this even further. Any standard deviation of match results above 0.5/sqrt(N) (as a fraction; 0.5*sqrt(N) as score points) is only possible is something very very fishy is going on. (In particular, the result of the previous game determining the result of the current one in whole or in part.)

[Alessandro Scotti]

Thanks for all the answers guys, I'm now all excited to test your suggestions but will have to wait a few more hours until this night!
Another tool I'm planning to use is a chess tree browser. I have missed one several times in the past and maybe now it's time to get one. IIRC Vasik uses Chant, while Anthony has written his own... the two strongest programs in the world use a tree browser, that must mean something!
Will post any news here of course, for good or bad!

[Guetti]

Thanks for all the answers guys, I'm now all excited to test your suggestions but will have to wait a few more hours until this night!
Another tool I'm planning to use is a chess tree browser. I have missed one several times in the past and maybe now it's time to get one. IIRC Vasik uses Chant, while Anthony has written his own... the two strongest programs in the world use a tree browser, that must mean something!
Will post any news here of course, for good or bad!

I didn't know about Chant! Thanks for mentioning it!


Edit: I see there is a linux version. Can it be run in OSX?

[hgm]

Another tool I'm planning to use is a chess tree browser. I have missed one several times in the past and maybe now it's time to get one.

In order to scrutinize a tree, the only fool-proof way I found was rerun the search from scratch for each node that you want to examine.

So basically you should clear the hash table before each search (or restore it to a known state), and then give the path of moves leading to the node you want to examine. In Search() you cab then print out any information you want (controlled by the setting of debug flags) if the actual path to it matches the requested one. E.g. print an overview of moves, and their search scores.

You can then step by step add plies to the path, for those moves that you think have an incorrect score. At some point the only thing the selected node prints is that it had a sufficient-draft hash hit. In that case you remember the entry and signature. When doing the re-search, you then use this (entry, signature) to set a kind of break point, that prints the path for you when there is a replacement in the entry with that signature.

That is really all that is needed: a test if you are in the node you are interested in, based on path or on the hash probe. and some output under the condition that there is a match.

[Spock]

Thanks for all the answers guys, I'm now all excited to test your suggestions

But I guess your wife is NOT excited....

[Alessandro Scotti]

Another tool I'm planning to use is a chess tree browser. I have missed one several times in the past and maybe now it's time to get one.

In order to scrutinize a tree, the only fool-proof way I found was rerun the search from scratch for each node that you want to examine.

So basically you should clear the hash table before each search (or restore it to a known state), and then give the path of moves leading to the node you want to examine. In Search() you cab then print out any information you want (controlled by the setting of debug flags) if the actual path to it matches the requested one. E.g. print an overview of moves, and their search scores.

You can then step by step add plies to the path, for those moves that you think have an incorrect score. At some point the only thing the selected node prints is that it had a sufficient-draft hash hit. In that case you remember the entry and signature. When doing the re-search, you then use this (entry, signature) to set a kind of break point, that prints the path for you when there is a replacement in the entry with that signature.

That is really all that is needed: a test if you are in the node you are interested in, based on path or on the hash probe. and some output under the condition that there is a match.

Well what I want to do is to dump the entire search tree to disk and then browse it with a (simple) user interface. This will be integrated into the engine so I can invoke evaluation, search, etc. from any position. It will also help setting those "path breakpoints" if needed.
Besides debugging, I'm having a really hard time visualizing how a search works, so I expect this tool will help me on this too.

[hgm]

Hmm, it sounds extremely slow to have to write the entire tree on disk.

Interacticvely walking a tree is indeed a very good way to get acquainted with how your search works. But I always did it in real time, rather than from a post-mortem dump.

[Alessandro Scotti]

A simple program that simulates writing 2 million nodes (20 bytes each) takes about 40 seconds to produce the 40 MB output. It will be much slower on the Mac Mini but still manageable I think.