Winboard state machine diagram

[Harald]

Hi

I am trying to understand the communication between Winboard and the chess engine a little better.

See: Chess Engine Communication Protocol
http://www.open-aurec.com/wbforum/WinBo ... -intf.html
and: WinBoard/XBoard protocol state diagram
https://www.chessprogramming.org/Chess_ ... n_Protocol

Sometimes the Winboard documentation is hard to understand and at some points
the Chess Engine Communication Protocol diagram is not right but it helps a lot.
I try to draw an own diagram with yEd Graph Editor misusing the flow chart symbols.
This (attached image) is what I got so far. It is not done yet and it may have errors.
I did not check the analyze part - only the thinking part.
Are there any comments or corrections?

The fat blue boxes are the states. The ellipses are Winboard commands.
The yellow boxes are engine responses. The small arrows indicate the possible
commands per state and the fat arrows are state changes.
The commands are grouped together in the order that Winboard uses.

Harald

[hgm]

I have always felt that the state diagram of CECP is not very useful / needlessly complicated because it is actually the 'product' of a number of independent or very weakly coupled simpler state machines. There is the ponder state, which is just on or off, affected only by the easy and hard commands (which do not affect anything else). Then there is the 'player state', which is white, black, none or analyze. This is affected by the commands new, go, playother, white, black, analyze, exit and force, while a game reaching a legal end also switches this state to 'none'. There is the game state, which is affected by new, variant, setboard, usermove, undo and remove, and by the engine moving when the thinking state times out (or, in some implementations, gets aborted). Finally there is the 'search state', which can be idle, thinking (= timed), pondering or analyzing. It is not really an independent state, but indirectly controlled by the states of the other state machines: as long as the player state is 'analyze' the engine will be analyzing the current position. If it is none, the engine will be idle. When it is white or black, the engine will be thinking as long as that color is on move, and idle or pondering (depending on the ponder state) for as long as the other color is on move (with the exception that the initial position is never pondered on).

Also note that ping can besent in any state, and will be processed (resulting in emission of 'pong') when its turn comes up in the order of received commands, i.e. when all commands received before have been executed. For commands that caused the engine to start thinking that includes the time the engine thinks and possibly prints its move. Pondering, however, should not be considered as being execution of any command, but just something the engine does as a pass time while waiting for a command (which then can be executed immediately).

BTW, WinBoard never sends a 'new' or a 'setboard' command durig analysis mode, as the 'New Game' menu item also switches the mode to 'machine black', and quits analysis first (through 'exit'). I don't really see why any GUI would feel the need to use these commands in analyze mode.

[Harald]

Thank you for that explanation. May be I try to visualize that somehow.
My diagram was created from the engine's point of view. While it is mostly
just waiting or looping around in the observing force mode it gets all kinds
of commands that just change some variables. Perhaps some member variables
of an Winboard class or a chess clock or a player status. but sometimes there is
a bigger change. A search is started (as a thread) the engine is thinking and has
to do a lot more things. There could be a global variable where the main command
loop and the search thread communicate and time out or stop the search.
Then it has to change state again, stop the search and continue in force mode
or pondering. The commands that can be received in these states or modes can differ.
There could be a check which command is allowed in each engine state.

At least there are some parts in the Chess Engine Communication Protocol documentation
that are just wrong like "new: ... Leave force mode ...". And the transition from
'observing' to 'waiting' in the WinBoard/XBoard protocol state diagram also does not
make much sense without further explanation.

Finally there is the Winboard GUI itself that send its commands in groups and patterns.
If the main loop of the engine does not want to recognize these groups it must at least
react to the individual commands at exactly the right moment when it receives them
from an input queue (another thread). These command groups also minimize the
possible GUI to engine commands in each situation. It would be nice to know for a new
engine programmer what to expect and what must be really implemented for a very
basic engine.

[hgm]

The command groups are usually work-arounds for bugs in some historic engines. (Which I consider a very bad idea.) But they were designed in such a way that they would do what was desired if all the commands in it would individually do what the specs say they should do. It is just that they also work in case some of the commands do wrong things.

Note that the specs hardly ever restrict when (i.e. in what state) a given command can be used, although it is true that many commands just make no sense in certain situations. The basic principle is that commands are forbidden during thinking (i.e. when the player state equals the side to move), except those that should stop the thinking (quit, force and ? = move now), so that it is OK (although not recommended) to abort thinking on reception of any command. Only a selected set of commands is allowed while analyzing, but this seems a pointless restriction, as the useful allowed commands (usermove, undo and exit) would always require the search to be stopped (and then possibly a new one to be started). So it is OK to have reception of any command just abort an analysis search (and then judge whether a new one should be started after the command is executed according to specs.) The specs necessitate a dependence on the current player mode of the execution of 'new': it should refrain from changing the player mode to 'black' when it is received in analyze mode. (But I don't think there is any GUI that would send it in analyze mode...) For the 'memory' command I specified it is not allowed during a game, but this was only to releave the engines from worrying whether they should somehow try to preserve the hash content during a size change, and make it OK to always clear the hash when the size actually has to be changed.

The 'leave force mode' phrase in the decription of the 'new' command is indeed a bit strange, as it suggests the command can only be issued in force mode (i.e. player mode 'none'). Which then is obviously contradicted (like that it should switch to play black) by the requirement that it should be allowed in analyze mode. Unless one would equate analyze mode with force mode, which seems wrong to me. (I.e., it would then group two player states that obviously require different behavior ('none' and 'analyze') under the same name 'force mode', which is just confusing.) WinBoard would send most commands only in player mode 'none', and there even has been a v3 specification of the protocol that would require that for many commands (such as setboard), but this has never been oficially adopted.

My preferred implementation of an engine's protocol driver keeps three state variables: engineSide (white/black/none/analyze), sideToMove (white/black) and pondering (on/off). At the top of the loop that reads and processes one command I then judge from the combined state what the engine should do: think if engineSide == sideToMove, ponder if pondering == on && engineSide == Opponent(sideToMove), or analyze if engineSide == analyze. The commands then just manipulate the states. An alternative (perhaps more logical) would be to recognize a third game state 'finished' (which should not equate to player state 'none'!), an switch to that after checkmate and such; then you would not have to switch the player mode to 'none' in these cases to make sure the engine starts searches in positions with no moves.

Code: Select all

#define WHITE 1
#define BLACK    2
#define NONE     0
#define ANALYZE  3
#define FINISHED 4

int stm = FINISHED;
int engineSide = NONE;
int pondering = FALSE;

while(1) {
  ReadAndExecuteOneCommand();
  
  if(engineSide == stm) Think(), MakeMove(), PrintMove();
  else if(engineSide == WHITE + BLACK - stm) Ponder();
  else if(engineSide == ANALYZE) Analyze();
}

I always felt the most confusing part of the specs was the handling of the clocks. The logical way to think of the remaining times would be as part of the game state (white time and black time), but they are actually described in the specs as 'my time' and 'opponent time'. So when the engine gets to play the opposite side as a result of one command (go after new), or a sequence of commands (new - force - usermove - usermove - go), you effectively swap the white and black times. (Which is of course always undesired, and thus forces the GUI to pre-emptively swap the times back by explicit time and otim commands before the command that sets the engine playing the other color than before.)

[xr_a_y]

Code: Select all

#define WHITE 1
#define BLACK    2
#define NONE     0
#define ANALYZE  3
#define FINISHED 4

int stm = FINISHED;
int engineSide = NONE;
int pondering = FALSE;

while(1) {
  ReadAndExecuteOneCommand();
  
  if(engineSide == stm) Think(), MakeMove(), PrintMove();
  else if(engineSide == WHITE + BLACK - stm) Ponder();
  else if(engineSide == ANALYZE) Analyze();
}

Are you sure this is working with pondering.
The ReadAndExecuteOneCommand() function is likely to wait for something on stand input and there shall be no input between the end of search and the moment where pondering should start. Am i wrong ?

[hgm]

You are right. The 'else' at the start of the Ponder() line should not be there.

Note that this assumes a search can be interrupted and then re-started without any ill effects, on a ponder hit. If that is not the case, speculative pondering should be implemented in a different way, which doesn't really distinguish thinking from a ponder search, as the latter can turn into the former on a ponder hit. So you would get something like

Code: Select all

while(engineSide == stm || engineSide == Opponent(stm) && pondering == ON && ponderMove != INVALID) {
  ponderSearch = (computer != stm);
  if(ponderSearch) MakeMove(ponderMove);
  Search(); // might clear 'ponderSearch' if ponder hit occurs
  if(ponderSearch) UnMake(ponderMove);
  else MakeMove(bestMove), PrintMove(bestMove);
}

[xr_a_y]

You are right. The 'else' at the start of the Ponder() line should not be there.

Ponder shall be started as soon as searched move is written to standard output then ?

Edit : I miss read your answer. Without the else, you are saying that search shall be synchronous ? This is not possible, some command can be received during search.

[hgm]

The ponder search is supposed to terminate if input arrives. With speculative pondering it should make an exception if the input is a 'usermove' command with the move it is pondering on; in that case it should change the search state from pondering to thinking, and continue searching.

(Note I had been editing something about speculative pondering to my previous posting.)

[Robert Pope]

The ponder search is supposed to terminate if input arrives. With speculative pondering it should make an exception if the input is a 'usermove' command with the move it is pondering on; in that case it should change the search state from pondering to thinking, and continue searching.

(Note I had been editing something about speculative pondering to my previous posting.)

Is that right? I thought ponder only got interrupted by specific commands, e.g. new or a move, not any input. If ping is sent, you don't stop pondering. You just respond and continue pondering.

[hgm]

OK, it is true I assumed here that the GUI would not send frivolous commands, as WinBoard indeed would not, but will only send something when there is a reason to stop pondering and start doing something else. You only ponder during a game, when the opponent is thinking, and the only thing you can expect is a move, draw offer or resign. In particular, there never is any reason to send 'ping' to a pondering engine. But it is true the protocol specs do not forbid this, so strictly speaking what I was saying is not correct. The incorrectness w.r.t. 'ping' would not hurt on any known GUI, though.

But this would not hold for the 'time' and 'otim' commands; they would always preceed 'usermove', which could be a ponder hit. In fact these commands have always caused me trouble in the particular implementation I always use for checking input. I guess that really is a consequence of failing to switch off all input buffering. What appears to happen then is that the 'otim' and 'usermove' command will get buffererd when I read the 'time' command, because WinBoard sends all these three at once. If you then process 'time', while continuing the ponder search, and keep polling for input, the 'otim' and 'usermove' will stay hidden for you when you poll for pending input. Because they are no longer in the pipe, but in the input buffer. Then things will hang.

So what I do is consider 'time' + 'otim' + 'usermove' a single (3-line) command, and always continue reading and processing commands after 'time' and 'otim' whenever these are received. I admit this is dirty and not robust; it counts on the GUI not sending spurious 'time'/ 'otim' commands. The robust solution would be to switch off all buffering, but I never learned how to do that, because this work-around had already solved it.

If the engine treats 'time' and 'otim' as separate commands, then it is true that these should always be processed during search, without causing it to abort. And that is really only needed when you do speculative pondering (so that you can have ponder hits), and the search is not 'self-resuming' when you restart after an abort.

Another command that should be processed during search without aborting it (but during analysis, rather than ponder) is '.' (the polling for periodic updates). Usually I also process the 'post' and 'nopost' commands during search. Most of this is moot, though; in practice these commands would never come during match conditions. And otherwise you would hardly care whether a ponder search is interrupted or not because the user is messing with the GUI menus to distract a pondering engine.

Now that I think about it, it might also be good to handle draw offers during search (by just setting a flag to indicate they have been made). So that next time you have to think you can consider accepting the draw as a possible root 'move'. In principle a draw offer could accompany a move that is a ponder hit. If you write your root search such that it always tries to accept a pending offer after every move it considered, it will not hurt if the offer arrives somewhere during the search.