Testing with Openings and "Repeat" Option: First white doesn't change.

[OliverBr]

Hello together,
when testing engines with an opening book, it's a good option to let each engine play an opening as white and black. This is what the "repeat" option in cutecess-cli is for.

So far, so good. But for a particular opening engine A always is white first, then black in the second game. Wouldn't it be better if this role changed, too? Something like:

Code: Select all

Game 1, Opening 1: Engine A - Engine B
Game 2, Opening 1: Engine B - Engine A
Game 3, Opening 2: Engine B - Engine A
Game 4, Opening 2: Engine A - Engine B
..

Background: After so much testing I have detected, that Engine B (the second one) always has an advantage. I think this is because of the hashtables, which are being filled from the previous game. Especially with the same opening this should be quite an advantage.

Perhaps the author of cutechess-cli may have a look into it. If there was an option for swapping the role of being white first, I would use it.

[brianr]

If the engines do not automatically clear the hash tables, you can use the restart=on cutechess-cli option.

[OliverBr]

If the engines do not automatically clear the hash tables, you can use the restart=on cutechess-cli option.

Currently OliThink clears the hashtable for a new game in analyze mode. I wasn't sure if it a good idea to clear it with every new game.
What do you think? Do other engines clear it with each game?

PS: If restart=on means it destroy the process then this is not a good option.

[brianr]

Clearing the hash table is up to the author.

In Tinker (which has not been worked on for several years), I clear the hash quite often. I don't clear it when in "force" mode (Tinker is CECP/Winboard, with minimal UCI support as an option). This allows the hash table to be used between moves when analyzing by hand. For test suite "analysis" mode, I think the hash table should be cleared.

With restart=on, killing the process takes very little extra time I have found. Some engines don't need it. In the early days of Leela chess, it was very important to do so, but it's caching is quite different from traditional hash tables anyway.

[Dann Corbit]

Clearing the hash for a new game is sensible. The old hash table will have nothing but crap in it, so why save it? You are not going to replay those moves.

When analyzing, clear the hash if the root move is not in the hash table.

A hash table full of the wrong stuff just causes more tests and moves and none of that stuff will matter.

I would never clear the hash during a single game

Also when analyzing, you are often analyzing a game. So don't clear the hasb analyzing unless the board suddenly changes and the current root is not found in the hash.

Just my raw, unmeasured opinion

[OliverBr]

Thanks for the input. The next version will clear the hash with any new game.

When analyzing, clear the hash if the root move is not in the hash table.

This is most interesting, too.

Something like this in search:

Code: Select all

struct entry he = hashDB[hp & HMASK];
if (he.key == hp) {
.....
} else if (!ply && analyze) memset(hashDB, 0, sizeof(hashDB));

With a new fen-position it is cleared anyway.