Using UCI to get deterministic output from engine

[gordonr]

Hi,

I'm writing a Java chess app that interfaces with a UCI engine. I have a series of test positions for which I want to get analysis from the engine. I want my tests to be reproducible so I'm trying to use the engine in a deterministic manner. So I'm using a single thread; a fixed depth search; and I clear the engine hash prior to starting any analysis.

However, I'm finding that if I analyse "test B" alone in a test run, it will fail. But if I run "test A followed by test B", both will pass. This is because analysing position B differs depending on whether position A was analysed previously or not.

- I don't restart the engine process between test positions. I assumed that a clear hash would be enough
- I may not be using the UCI "isready" in the right places. I only do that once when initially starting the engine; not between every test position

UCI commands prior to first analysis:

Code: Select all

UCI COMMAND [uci]
UCI COMMAND [isready]
UCI COMMAND [setoption name hash value 64]
UCI COMMAND [setoption name threads value 1]
UCI COMMAND [ucinewgame]
UCI COMMAND [position fen r4rk1/pppbqppp/5n2/3P4/3P4/b4B2/P1Q2PPP/RN3RK1 w - - 0 1]
UCI COMMAND [setoption name Clear Hash]
UCI COMMAND [setoption name Ponder value false]
UCI COMMAND [go depth 10 ]

UCI commands between each test position thereafter (the fen would change per test):

Code: Select all

UCI COMMAND [stop]
UCI COMMAND [position fen r4rk1/pppbqppp/5n2/3P4/3P4/b1N2B2/P1Q2PPP/R4RK1 b - - 0 1]
UCI COMMAND [setoption name Clear Hash]
UCI COMMAND [setoption name Ponder value false]
UCI COMMAND [go depth 10 ]

Am I missing a UCI "isready" somewhere? Maybe my Java code is executing some commands too fast in succession for the engine. Or something entirely else?

Thanks for any help,

Gordon

[hgm]

Perhaps the engine you are using randomizes intentionally. What engine are you using anyway? Does it support the Clear Hash option?

[F. Bluemers]

The engine maybe does not clear the killer and history tables after receiving clear hash.

[Evert]

I think you're supposed to send "ucinewgame" if the positions are unrelated. Otherwise the engine may not clear the history/killer tables.

[gordonr]

I've tried with Komodo CCT, Komodo 9.3 and a recent build of Stockfish to help rule out an engine specific behaviour.

I'll try inserting a "ucinewgame" between tests.

Thanks everyone for the suggestions.

[elcabesa]

If you want to be sure there is only one way. Close and reopen the engine between any search.
There are lot of structure engine didn't clear between searches. And I don't think uci new game will clear them all. (Killers, counterfoils history and so on)

[hgm]

Indeed, there is no guarantee that 'ucinewgame' will clear everything, or even anything. It is not really harmful to start a new game with a left-over history table (or a transposition table or hash table, for that matter) from a previous game. So why would they clear it?

[Evert]

If you want to be sure there is only one way. Close and reopen the engine between any search.
There are lot of structure engine didn't clear between searches. And I don't think uci new game will clear them all. (Killers, counterfoils history and so on)

There is indeed nothing that requires it. If it's done at all, then "ucinewgame" is your best bet short of restarting the engine (which would obviously always work).

[gordonr]

Thanks everyone for the advice.

Since I want my app to work with as many UCI engines as possible, I won't rely on "ucinewgame" even if it does work with my chosen engines. I will use an engine restart when I want reproducible results.

[bob]

Hi,

I'm writing a Java chess app that interfaces with a UCI engine. I have a series of test positions for which I want to get analysis from the engine. I want my tests to be reproducible so I'm trying to use the engine in a deterministic manner. So I'm using a single thread; a fixed depth search; and I clear the engine hash prior to starting any analysis.

However, I'm finding that if I analyse "test B" alone in a test run, it will fail. But if I run "test A followed by test B", both will pass. This is because analysing position B differs depending on whether position A was analysed previously or not.

- I don't restart the engine process between test positions. I assumed that a clear hash would be enough
- I may not be using the UCI "isready" in the right places. I only do that once when initially starting the engine; not between every test position

UCI commands prior to first analysis:

Code: Select all

UCI COMMAND [uci]
UCI COMMAND [isready]
UCI COMMAND [setoption name hash value 64]
UCI COMMAND [setoption name threads value 1]
UCI COMMAND [ucinewgame]
UCI COMMAND [position fen r4rk1/pppbqppp/5n2/3P4/3P4/b4B2/P1Q2PPP/RN3RK1 w - - 0 1]
UCI COMMAND [setoption name Clear Hash]
UCI COMMAND [setoption name Ponder value false]
UCI COMMAND [go depth 10 ]

UCI commands between each test position thereafter (the fen would change per test):

Code: Select all

UCI COMMAND [stop]
UCI COMMAND [position fen r4rk1/pppbqppp/5n2/3P4/3P4/b1N2B2/P1Q2PPP/R4RK1 b - - 0 1]
UCI COMMAND [setoption name Clear Hash]
UCI COMMAND [setoption name Ponder value false]
UCI COMMAND [go depth 10 ]

Am I missing a UCI "isready" somewhere? Maybe my Java code is executing some commands too fast in succession for the engine. Or something entirely else?

Thanks for any help,

Gordon

Clearing hash is not even close to being enough. You need to clear primary hash, clear pawn hash, clear eval hash, clear history counters, clear killer moves, clear counter-moves, i.e. clear EVERYTHING that has any influence on move ordering. If you miss anything, the node counts will change and the results can change.