Question

[bob]

Hi

Thank you for the response.

It' Ok with option depth = <arg>, but for me it is amazing that almost 100 % of games differ, with only some ms per move

good evening

Here's an even more fun experiment to try. Take two programs that let you specify a specific number of nodes per search, and try something like sn=1000000, and play a game. (1 million nodes per move). Then play another game with sn=1000001. It is a near-certainty that you won't get the same game. Try sn=1000002. Another new game. It was hard to imagine that one extra node per search would cause a game to diverge and follow two distinct paths. Yet it happens. When I had first discovered this and reported it here, someone tried that test 100 times, and out of 100 games, they produced one pair that duplicated every move... which is to me amazing.

[Milos]

That will repeat, for sure, but it gives wrong results. You won't know which of the two programs is better, since N plies to one program is not the same as N plies to another. If you try to measure improvements, any change that makes your tree larger (more extensions, fewer reductions, etc) or any change that makes you slower (more eval, more complex tests in search) will look good because with fixed depth, the slower search speed will not hurt you because you always go to the same depth regardless of speed...

Bob, you are absolutely right. But sometimes the tests at a fixed depth are useful.

Yes they are. I use 'em for debugging all the time, particularly where I have an unexpected crash that is infrequent. Because with fixed depth, I can always replay to the same point and get the same crash every time. It is pretty hard to find bugs if you can't recreate them on demand...

Actually searching at the fixed ply is extremely useful if you are optimizing you compile, optimizing different function implementations, e.g. want to check which popcnt implementation is the best. The procedure is very simple. Take few test positions, and measure the time to reach a certain ply. The engine which reaches the ply faster will always have higher ELO. Of course, this only works if you didn't touch search and evaluation, i.e. the pv line remains the same.

[bob]

That will repeat, for sure, but it gives wrong results. You won't know which of the two programs is better, since N plies to one program is not the same as N plies to another. If you try to measure improvements, any change that makes your tree larger (more extensions, fewer reductions, etc) or any change that makes you slower (more eval, more complex tests in search) will look good because with fixed depth, the slower search speed will not hurt you because you always go to the same depth regardless of speed...

Bob, you are absolutely right. But sometimes the tests at a fixed depth are useful.

Yes they are. I use 'em for debugging all the time, particularly where I have an unexpected crash that is infrequent. Because with fixed depth, I can always replay to the same point and get the same crash every time. It is pretty hard to find bugs if you can't recreate them on demand...

Actually searching at the fixed ply is extremely useful if you are optimizing you compile, optimizing different function implementations, e.g. want to check which popcnt implementation is the best. The procedure is very simple. Take few test positions, and measure the time to reach a certain ply. The engine which reaches the ply faster will always have higher ELO. Of course, this only works if you didn't touch search and evaluation, i.e. the pv line remains the same.

I don't run test matches for that kind of changes. I just use a set of positions searched to a fixed depth and compare speeds (NPS since I am not changing the shape of the tree with pure mechanical optimizations). If I change anything to do with the tree itself (evaluation, search, ordering, etc) I play matches using time limits.

The testing being mentioned here was playing full games as part of a match. Different animal from just testing a position to a fixed depth.