cutechess-cli: not restarting an engine because of tt

[lucasart]

Usually I use cutechess-cli for testing (because it is command line so I can easily start it on all nodes of the test cluster) but using strace I saw that it restarts every program after an iteration which gives me a clean transposition table every time.

Cutechess CLI does NOT restart engines, if you use it correctly...

It just sends "ucinewgame" between games. If you want to clear HT between games, that's where you should do it: upon receipt of "ucinewgame".

Can you show me a correct example?

I usually do this:

Code: Select all

cutechess-cli \
        -engine cmd=/usr/local/bin/dorpsgek proto=xboard name=dorpsgek \
        -engine cmd=./E_trunk.sh proto=uci name=trunk \
        -engine cmd=./E_trunk-age.sh proto=uci name=trunk-age \
        -engine cmd=./E_4171.sh proto=uci name=4171 \
        -engine cmd=./E_4173.sh proto=uci name=4173 \
        -engine cmd=./E_4182.sh proto=uci name=4182 \
        -engine cmd=./E_4183.sh proto=uci name=4183 \
        -concurrency $CONC \
        -each dir=$DIR tc=40/10+0.25 -rounds 500 -recover -repeat -tournament gauntlet -pgnout out/$HOST.pgn -site "$HOST" | tee log/$HOST.log

From the manual:

Code: Select all

  -games N		Play N games per encounter. This value should be set to
			an even number in tournaments with more than two players
			to make sure that each player plays an equal number of
			games with white and black pieces.
  -rounds N		Multiply the number of rounds to play by N.
			For two-player tournaments this option should be used
			to set the total number of games to play.

I'm not even sure using repeat makes any sense if you use games 1 (implicitely) as you do here.

[Ras]

It just sends "ucinewgame" between games. If you want to clear HT between games, that's where you should do it: upon receipt of "ucinewgame".

That command is not mandatory, so you should not rely on it.

[Ras]

This would only work when the engine plays itself, right?

Yes, but when the engine doesn't play (equivalent to "force" mode in Xboard), the hash tables are not used anyway.

If it is actively playing against an opponent, it works. All it does is checking that the position before the opponent's answer move is the same as when the engine has given its previous ply.

And it doesn't work at all when the engine ponders.

I think it does. You get a ponder move in UCI, and it doesn't matter whether you start the calculations on "ponder" or on "ponderhit" because the transmitted move list doesn't change.

Why would you want to clear the TT on a new game anyway?

Makes life easier for testing because things are reproducible after starting the engine. Looks like Folkert has that issue.

Empty entries are not more helpful than entires from another game.

They are because they offer themselves for being written without further considerations.

Aging would make the entries from the previous search be considered empty pretty quickly anyway.

Leaves still the testing issue.

I for my part havn't even implemented aging, mostly because there's no space for aging entries on my embedded platform. I'm doing it the hard way by deleting 25% of the entries before each move. That is, every 4th entry, with the offset from the start varying between 0 and 3.

That would prevent clearing of tables on a ponder miss, or a takeback.

Of course, that's an improvement.

[hgm]

This would only work when the engine plays itself, right?

Yes, but when the engine doesn't play (equivalent to "force" mode in Xboard), the hash tables are not used anyway.

If it is actively playing against an opponent, it works. All it does is checking that the position before the opponent's answer move is the same as when the engine has given its previous ply.

I don't think it does. When the engine plays against a human with ponder off, the subsequent 'go' commands it receives would come from positions set up by two more ply: the engine's previous move, and the human's reply. So the ply counter would have incremented by 2, not by 1. The engine is never fed the position where the opponent has the move.

And it doesn't work at all when the engine ponders.

I think it does. You get a ponder move in UCI, and it doesn't matter whether you start the calculations on "ponder" or on "ponderhit" because the transmitted move list doesn't change.

The ponder search would be started on a position that was set up with a speculated opponent move. This also adds two ply to the previous search: the negines own move that was a result from that search, plus the speculated move. After a ponder miss a search would be restarted on the true opponent move. This would now have as many ply as the previous (ponder) search, and the last ply would differ between the cases. In no case you would get a ply difference of one.

Why would you want to clear the TT on a new game anyway?

Makes life easier for testing because things are reproducible after starting the engine. Looks like Folkert has that issue.

For testing it would be better to clear hash after every move. Usually unexplained blunders that require debugging don't occur in the starting position. Most engines have a Clear Hash option, which would allow this to be contrulled by the developer that is debugging it. No need to automate that for normal users that just use it for playing. Besides, you could clear the hash on 'ucinewgame'.

Empty entries are not more helpful than entires from another game.

They are because they offer themselves for being written without further considerations.

Aged entries can also be written without further consideration. In fact it is an extra consideration to wonder whether the entry is empty, which is overhead during most part of the game, where empty entries have long since been used up.

I for my part havn't even implemented aging, mostly because there's no space for aging entries on my embedded platform. I'm doing it the hard way by deleting 25% of the entries before each move. That is, every 4th entry, with the offset from the start varying between 0 and 3.

Sounds very sub-optimal. Why not use under-cut replacement during the search, to have some protection for large drafts, and still prevent they conquer the entire table?

[Ras]

subsequent 'go' commands it receives would come from positions set up by two more ply

Ah, there's the gap: I'm storing the hash value from after the engine transmitted its move with "bestmove". That's how I get one ply difference.

Besides, you could clear the hash on 'ucinewgame'.

That's not mandatory, so you can't rely on it.

Aged entries can also be written without further consideration.

Wondering how to notice the age? If it's in terms of "move number in the game", then with a new game, the moves will look like far in the future?

Sounds very sub-optimal.

It deals with aging without having to store information. After four moves, I can be sure that no old entries will waste space. Besides, that's only done if the hash table usage is above 70% when starting a new calculation, so something has to go anyway.

Why not use under-cut replacement during the search, to have some protection for large drafts, and still prevent they conquer the entire table?

Uhm.. I guess I'm lacking knowledge to even remotely understand the question (which may be part of the answer as to why not).

Anyway, the actual problem is that with 112k of hash tables, there isn't much point in better replacement schemes. Sure, the PC version has way more, but I want to keep the code mostly in line.

[hgm]

Ah, there's the gap: I'm storing the hash value from after the engine transmitted its move with "bestmove". That's how I get one ply difference.

Ah OK.

Besides, you could clear the hash on 'ucinewgame'.

That's not mandatory, so you can't rely on it.

If you are a developer that is interested in reproducing errors, you would make sure you use a GUI that send it, or operate the Clear Hash button. If you are a user, you would not care.

Wondering how to notice the age? If it's in terms of "move number in the game", then with a new game, the moves will look like far in the future?

I usually keep a2 or 3-bit counter that is incremented every time the engine thinks, and store that in the age field of the entry when it is successfully probed or written. There is no need to reset the counter when a new game starts.

It deals with aging without having to store information.

Yes, that is an advantage. But you indiscriminately throw away valuable and almost worthless entries, and that is not good. Do you really have no bits to spare? If you erase before a search, even a single bit, toggled each new search would already be good enogh to discriminate between entries that were used in the last search, ad entries that were not (and thus can probably be safely erased).

Anyway, the actual problem is that with 112k of hash tables, there isn't much point in better replacement schemes. Sure, the PC version has way more, but I want to keep the code mostly in line.

The smaller the hash table, the more you can gain by using it efficiently.

[Ras]

If you are a user, you would not care.

That's the problem - if you get user bug reports, they are harder to evaluate.

I usually keep a2 or 3-bit counter that is incremented every time the engine thinks

That's cute!

Do you really have no bits to spare?

Just reviewed the data structure. A uint32_t for the move, uint32_t for the upper half of the position hash, int16_t for the value.

uint8_t for the alpha/beta/exact/empty flag, but the free 6 bits are already used for fiddling in 6 more position hash bits so that together with the implicit index bits, I get up to 48 bits. (the pawn hash table is even worse than that, not to mention the opening book format.)

But the depth is an int8_t, and even in the PC version, the depth is limited to 40 plies. That would leave two bits, sufficient for the counter!

Looks like an interesting idea, thanks.

[AlvaroBegue]

Do you really have no bits to spare?

Just reviewed the data structure. A uint32_t for the move [...]

32 bits for a move is very wasteful. I use 16: 6 for the `from' field, 6 for the `to' field and 4 for the `type' field (normal, castling king side, promotion to queen, promotion to knight, en-passant capture...). See here: https://chessprogramming.wikispaces.com/Encoding+Moves

[Ras]

32 bits for a move is very wasteful.

Woah, that's something I've managed to completely overlook until now. Talk of taking structures as given. :-/

That alone would shrink my main hash tables from 96k to 80k if I split up the upper position hash value into two uint16_t to deal with alignment issues.

The pawn/rook hash table is 20k, and the linker still shows 7k free RAM. This means I might be able to double my pawn/rook hash table! Just have to shove around lots of stuff because the RAM areas aren't continuous. But the bigger one is 128k, and 80k plus (then) 40k would fit in.

That could speed up things because my pawn eval is quite expensive.

Thanks!

[hgm]

I certainly would not waste precious bits on extending the signature. Stockfish gets away with a 16-bit signature! For the small hash tables you can afford, compactness of the entries is a major concern.

Note that for encoding Chess moves in a not-very-cumbersome way 13 bits would be sufficient: from-square, to-square and a single bit for flagging special moves. When that bit is set, the to-square can store an index to a table that decodes the special move, and contains the move step (so the true to-square can be calsulated from the from-square), move type (promotion, e.p. capture, castling, double-push), as well as any additional information needed to specify a move of that type (promotion piece, relative location of e.p. victim, rook from- and to-squares, e.p. square). Because I typically use 0x88-type square-number encoding, the '8' bit offers a natural location to hide a flag. E.g. in the micro-Max hash I use the 0x88 bits in the from-square byte as the bound-type flas. The '8' bit in the to-square could be the special-move flag, and the 0x80 bit the age flag. In addition you would need 16 bit score, 8 bit depth, 32-bit signature, to give 9-byte entries, 7 of which you could put in a 64-byte bucket, with still one byte to spare. You could put an in-check flag there. (But you could also borrow a bit of the depth field for that; allowing the depth to run up to 256 seems excessive.)

The point is that it is quite cheap to mask all the flag bits out of numeric information (like square numbers), and that for testing specific flags you don't care what is in the rest of the word anyway. So you don't really suffer significant overhead from packing / unpacking the hash entries. The micro-Max test for usable bound type would look exactly the same when the bound-type flags had been in a word of their own, rather than packed with the from-square:

if((fromSqr & 0x80 || score <= alpha) && (fromSqr & 8 || score >= beta))

You could even consider using a 24-bit signature, so that you can pack 8 entries in a 64-byte cache line.

I think that in principle it would be possible to pack a move in 8 bits, for orhodox Chess. But then you would need to encode and decode it through tables, in an non-trivial way. I think SCID does that, for its database fomat.