WinBoard and EPD suites

[hgm]

Someone suggested to me that it would be useful to have WinBoard handle EPD suites. Now I have never done any EPD suites, so I only have a vague idea of what that entails. My question therefor is: what should WinBoard actually do.

Currently it can already read a file with FEN or EPD (-loadPositionFile), and step automatically through them (-loadPositionIndex -1). But it only does that for playing games from these starting positions. It would of course be only a minor modification to limit those 'games' to a single move only, before restarting the next.

So the main question is, what should WinBoard record and report from this. From full games in match mode it would count the wins/draws/losses, and display those as a popup at the end. What would we want here? Record the time until a given best move is first found, and calculate the average? Record the moves that are found after a preset time, and count the good solutions? How are the engines to be run in this mode? Just set them for a long time, and use the move-now command after the time is up? (But what if they do not support it?) Use the st command to set them for fixed time per move? (But what if they then think much shorter than this, to keep a safety margin?)

With one-line reporting for each position, (which could be done, for instance, in the lower pane of the Engine-Output window, like in -autoKibitz mode), what should the line preferably contain? Another EPD?

Any suggestions would be most welcome.

[Edmund]

The first challenge is to read the solution from the epd string; some variations exist in the testsuits: some only have one bestmove, some have several bestmoves, some report weights for several (good) moves, others have an "avoid move"

Next question is what statistics to collect:
usually the following information is reported
a) number of correct solutions found in the given time per move
b) total time taken to find the best moves
- when the best move is found but then search changes its mind and eventually comes back to the best move; the time of the final change is accounted
- care has to be take in case of multiple best moves
- if the bestmove was not found the total searching time is added
c) individual epd statistics (score when found, time taken, depth)

all of this should preferable be copy&paste friendly so that one can easily move it to a spreadsheet and compare the data with other results.

some tools offer the feature to stop the search if the best move has been reported for the last n depths

[Stephan Vermeire (Brutus)]

I fully agree to the suggestins of Edmund above!
You should have a look at Arena too, it has some sort of epd-testing module incorporated.

Stephan

[hgm]

The first challenge is to read the solution from the epd string; some variations exist in the testsuits: some only have one bestmove, some have several bestmoves, some report weights for several (good) moves, others have an "avoid move"

Well, that does not sound realy difficult. I suppose the status is 'solved' as long s the last given move by the engine is not amongst the avoid moves, and when there are no avoid moves, when it is amongst the bestmoves. Or should there be a three-state scoring (like win/draw/loss), where a loss would be an avoid move, and a win a bestmove, an otherwise draw?

Next question is what statistics to collect:
usually the following information is reported
a) number of correct solutions found in the given time per move
b) total time taken to find the best moves
- when the best move is found but then search changes its mind and eventually comes back to the best move; the time of the final change is accounted
- care has to be take in case of multiple best moves
- if the bestmove was not found the total searching time is added
c) individual epd statistics (score when found, time taken, depth)

all of this should preferable be copy&paste friendly so that one can easily move it to a spreadsheet and compare the data with other results.

some tools offer the feature to stop the search if the best move has been reported for the last n depths

OK, all that sounds feasible; I can separate output fields in the Engine-Output window by tabs. What I still wonder about is how the engine should be treated. I.e. under which time control they should be run, if move-now is to be used, etc.

[Kempelen]

The first challenge is to read the solution from the epd string; some variations exist in the testsuits: some only have one bestmove, some have several bestmoves, some report weights for several (good) moves, others have an "avoid move"

Well, that does not sound realy difficult. I suppose the status is 'solved' as long s the last given move by the engine is not amongst the avoid moves, and when there are no avoid moves, when it is amongst the bestmoves. Or should there be a three-state scoring (like win/draw/loss), where a loss would be an avoid move, and a win a bestmove, an otherwise draw?

Well, actually deciding when a problem is solved is very inconclusive. A move can be choosen and the engine change it the next iteration, maybe because it has not found the correct idea.
One solution is to consider problem solved when the move has been for two iterations without change mind, and also consider the solution total time until first appear of the correct move.... and this could also be incompatible if there is a time-out and i.e. the correct move has been chosen in last 5 secs.

[Michel]

For what it is worth. Some people use polyglot to do epd-tests. These are the options
provided by polyglot (from the man page)

polyglot epd-test
(possibly with a config file as first argument) PolyGlot supports
besides the generic options described above the following additional
options.

-max-depth (default: 63)
Unconditionally stop the search when this depth has been reached.

-max-time (default: 5.0)
Unconditionally stop the seach after this amount of time.

-depth-delta (default: 3)
Stop the search if the solution as been found and the best move has
been constant for this many depths, on condition that the mininal
depth and minimal time have been reached.

-min-depth (default: 8)
Minimal search depth when the search is stopped using
"-depth-delta".

-min-time (default: 1.0)
Minimal search time when the search is stopped using
"-depth-delta".

[hgm]

Well, that does not help WinBoard engines much... I guess UCI engines always obey the stop command, to implement pondering.

-max-depth and -max-time correspond to the already-existing WinBoard -depth and -st options. For the others new options coud be made, or the time-control options -mps and -tc could be requisitioned for this. Or -adjudicateLossMoves, although I don't think that is an option (but internally fixed to 3). But it could be made into one.

[Michel]

Sorry you misunderstood my point. I thought it would be useful to give an explicit example of how epd-testing can be implemented.

[hgm]

Yes, you are right; this is helpful in deciding how to extract data. I was still focused on how to control the engine, though, which seems a much bigger problem than in UCI.

[Michel]

I was still focused on how to control the engine

Well stopping the search early is optional of course but it really speeds up testing.

For engines that implement "?" (move now) stopping early is not a problem.

EDIT.

And of course analysis mode can also be used. One can always exit analysis using
the "exit" command.