xboard (HGM or others)

[bob]

I was playing some games with xboard this afternoon vs gnu chess 5.50 (discussion on Rybka Forum) and when I got to the point of +20 -7 I stopped the match. When I looked, 5 of those 7 losses were on time. the time control I was using was 30 seconds for 60 moves + 0.1s increment. I have been running a lot of cluster testing at a faster time control than that and absolutely NEVER lose a game on time.

Here is a sample of the quirk I am seeing:

time remaining: 30.14 (Crafty).
Black(3): otim 3012
time remaining: 30.12 (opponent).
Black(3): c6

At this point black has played a move, and Crafty has 30.14 seconds on the clock. Crafty is in book, and moves in what appears to me to be instantaneous form, and I then see this:

White(4): Nf3
time used: 0.00

which is what I expected, it took zero time to play that...

Then I see this:

Black(4): time 3016
time remaining: 30.16 (Crafty).
Black(4): otim 3015
time remaining: 30.15 (opponent).
Black(4): d5

IE 0.06 seconds were burned (if it really took 0 time I would expect the time to go from 3014 to 3024 since the inc is .1 (or 10 clock units). Is there some unexpected lag in xboard that I am not aware of? This is on a dual-cpu core-2 I7 2.0ghz box. I played 100 games with my referee program (no overhead to speak of) and there were zero time losses. Not critical, but it is interesting. Any ideas???

No way a book move that was correctly pondered, using an SSD for the book, is going to take any measurable time.

The game went 6 moves in book before gnu went out, when Crafty started the first search out of book, the time was 30.09, meaning the 0.1 sec inc and just about been burned up while in book. Seems strange to me. I also see quirks like this throughout the game where time is stepping along maybe 10 units of clock time faster than I expect inside crafty..

I don't have animation running, but do have a GUI up with engine thinking window open. Perhaps too much overhead???

[hgm]

Well, it is hard to say without having the xboard.debug file (made by XBoard when you run it with the option -debug), which lists all the engine communication with time stamps.

Move animation can cause problems like this, but you have that switched off.

It could be system overload. I don't know at what priorities you run the GUI and engine processes. XBoard has a -niceEngines option that reduces engine priority to make sure the engines cannot compete the GUI out of CPU time, and IIRC the default value for is was set to 'nice 10' the engines. Of course XBoard has to update the display and the Engine Output, but I cannot imagine this would take on the order of 100 ms. (It does not redraw the entire display, just the squares that changed or where touched by highlight arrows.)

I noticed that sometimes updating partially covered text windows when they are scrolling takes a lot of time.

[matthewlai]

I was playing some games with xboard this afternoon vs gnu chess 5.50 (discussion on Rybka Forum) and when I got to the point of +20 -7 I stopped the match. When I looked, 5 of those 7 losses were on time. the time control I was using was 30 seconds for 60 moves + 0.1s increment. I have been running a lot of cluster testing at a faster time control than that and absolutely NEVER lose a game on time.

Here is a sample of the quirk I am seeing:

time remaining: 30.14 (Crafty).
Black(3): otim 3012
time remaining: 30.12 (opponent).
Black(3): c6

At this point black has played a move, and Crafty has 30.14 seconds on the clock. Crafty is in book, and moves in what appears to me to be instantaneous form, and I then see this:

White(4): Nf3
time used: 0.00

which is what I expected, it took zero time to play that...

Then I see this:

Black(4): time 3016
time remaining: 30.16 (Crafty).
Black(4): otim 3015
time remaining: 30.15 (opponent).
Black(4): d5

IE 0.06 seconds were burned (if it really took 0 time I would expect the time to go from 3014 to 3024 since the inc is .1 (or 10 clock units). Is there some unexpected lag in xboard that I am not aware of? This is on a dual-cpu core-2 I7 2.0ghz box. I played 100 games with my referee program (no overhead to speak of) and there were zero time losses. Not critical, but it is interesting. Any ideas???

No way a book move that was correctly pondered, using an SSD for the book, is going to take any measurable time.

The game went 6 moves in book before gnu went out, when Crafty started the first search out of book, the time was 30.09, meaning the 0.1 sec inc and just about been burned up while in book. Seems strange to me. I also see quirks like this throughout the game where time is stepping along maybe 10 units of clock time faster than I expect inside crafty..

I don't have animation running, but do have a GUI up with engine thinking window open. Perhaps too much overhead???

I have also played thousands of games at 1s + 0.1s inc with GNU Chess 5.50 using my own referee program, and there were either no time losses or <10 time losses (don't remember).

[hgm]

Does your own referee program draw a graphical board and display the engine output on the screen in a scrolling window?

[matthewlai]

Does your own referee program draw a graphical board and display the engine output on the screen in a scrolling window?

Nope it doesn't. It's pure command line.

I was just confirming that it's not a problem with the engine.

I had similar problem with xboard running very fast games in the past, too, and had to remove all GUI code (this is way before noGUI).

In the case of xboard, would it be possible to stop both engines' clocks while the UI updates are happening?

I doubt it actually matters with my command line referee program, but the very first thing I do when I receive a move from an engine is to stop the clock. Then I do whatever I need to do, and don't start the opponent's clock until after the move has been given to the opponent.

[hgm]

In the case of xboard, would it be possible to stop both engines' clocks while the UI updates are happening?

I doubt it actually matters with my command line referee program, but the very first thing I do when I receive a move from an engine is to stop the clock. Then I do whatever I need to do, and don't start the opponent's clock until after the move has been given to the opponent.

In principle this would be possible. XBoard also stops the clock immediately, but this automatically starts the opponent's clock.

Waiting for the GUI to complete all its tasks is not ideal, because it does away with the parallalism that occurs when the GUI does its tasks while the engine is already thinking. Note that the most time-consuming part of the GUI is simply waiting until it is time to display the next animation frame. Stopping both clocks is also undesirable in ponder games.

What you really would want is to catch the case where the engine already sends a move before the GUI has completed its tasks for the previous one. So you would want to be able to peek at new input while still in the event handler for previous input, just to time-stamp all the input. Then you could use the time-stamps to update the clocks, rather than the time when you process the messages.

One problem is that engines might sends lot of thinking output before they come with the actual move that would 'press their clock'. You would have to buffer all that.

[matthewlai]

In the case of xboard, would it be possible to stop both engines' clocks while the UI updates are happening?

I doubt it actually matters with my command line referee program, but the very first thing I do when I receive a move from an engine is to stop the clock. Then I do whatever I need to do, and don't start the opponent's clock until after the move has been given to the opponent.

In principle this would be possible. XBoard also stops the clock immediately, but this automatically starts the opponent's clock.

Waiting for the GUI to complete all its tasks is not ideal, because it does away with the parallalism that occurs when the GUI does its tasks while the engine is already thinking. Note that the most time-consuming part of the GUI is simply waiting until it is time to display the next animation frame. Stopping both clocks is also undesirable in ponder games.

What you really would want is to catch the case where the engine already sends a move before the GUI has completed its tasks for the previous one. So you would want to be able to peek at new input while still in the event handler for previous input, just to time-stamp all the input. Then you could use the time-stamps to update the clocks, rather than the time when you process the messages.

One problem is that engines might sends lot of thinking output before they come with the actual move that would 'press their clock'. You would have to buffer all that.

If it needs to wait until the next vsync, that would explain the delay. VSYNC is at 60 Hz, so the average delay would be about 8ms. That is a lot of delay for 1s+0.1s games.

Timestamping would be ideal, but also very difficult to implement, and you would still be giving the opponent extra pondering time.

Stopping both clocks is very easy, and I believe is much fairer than the current approach (where engine's clocks are started before they are even given the move!).

I have no idea what the xboard architecture looks like but it may be beneficial to do all the time critical stuff in a separate non-GUI thread, and just send async messages to the GUI?

[hgm]

XBoard is a single-threaded program, where the main loop of the GUI library dispatches the events one by one. So during processing of each event no other events are seen; you have to return first to the main loop to retrieve and dispatch the next event. (Unless you would call the main loop recursively from your event handler.)

An alternative would indeed be to have different threads for each engine, which do blocking input, and process that input when it arrives. But this immediately leads to all kind of concurrency problems. You cannot start updating the board display when the previous move is still busy doing that, for instance.

Currently input events call a handler 'HandleMachineMove', which checks the move for legality, sends it to the opponent, switches the clocks and then starts the GUI business. The non-GUI tasks could be split off and handled by the engine input threads, which then pass the command through an internal pipe to the GUI thread that does the GUI business just as XBoard does it now (except for manipulating the clocks and passing on the move, which the GUI knows is already done for the move by the time it receives it). One should just hope that the internal pipe will never fill up, or the engine threads would block trying to write on it. So basically XBoard would start up 'semi-adapters' as threads connected to it by pipes for the incoming traffic only, just like it now starts up Polyglot as an external process.

What I am afraid of, though, is this: this whole idea of threads would work very nice if there was an infinite number of cores available. In practice people might not even reserve a single core/HT for the GUI, so that all these threads do in practice compete for CPU amongst each other and with the engines. So that they would still wait for each other to complete their task and release the CPU by blocking for input before the OS will start another one. So unless you have a really smart OS and clever assignement of thread priorities, it is questionable whether you really gain anything. The OS should really suspend the GUI thread or engines immediately when input arrives on an engine thread (and no idle cores are available), and suspend the engines as soon as the GUI thread wants to do something.

[matthewlai]

An alternative would indeed be to have different threads for each engine, which do blocking input, and process that input when it arrives. But this immediately leads to all kind of concurrency problems. You cannot start updating the board display when the previous move is still busy doing that, for instance.

Using threads would always bring complications, but I don't think this example is particularly difficult to solve.

For example, the GUI thread and the engine threads can share a (properly protected) FIFO (like an event queue) like you mentioned.

Currently input events call a handler 'HandleMachineMove', which checks the move for legality, sends it to the opponent, switches the clocks and then starts the GUI business. The non-GUI tasks could be split off and handled by the engine input threads, which then pass the command through an internal pipe to the GUI thread that does the GUI business just as XBoard does it now (except for manipulating the clocks and passing on the move, which the GUI knows is already done for the move by the time it receives it). One should just hope that the internal pipe will never fill up, or the engine threads would block trying to write on it. So basically XBoard would start up 'semi-adapters' as threads connected to it by pipes for the incoming traffic only, just like it now starts up Polyglot as an external process.

That is exactly what I had in mind as well. If the pipes ever fill up, that means xboard is not drawing fast enough, and that would be a separate problem I believe.

What I am afraid of, though, is this: this whole idea of threads would work very nice if there was an infinite number of cores available. In practice people might not even reserve a single core/HT for the GUI, so that all these threads do in practice compete for CPU amongst each other and with the engines. So that they would still wait for each other to complete their task and release the CPU by blocking for input before the OS will start another one. So unless you have a really smart OS and clever assignement of thread priorities, it is questionable whether you really gain anything. The OS should really suspend the GUI thread or engines immediately when input arrives on an engine thread (and no idle cores are available), and suspend the engines as soon as the GUI thread wants to do something.

That is true. It's an impossible problem to solve. If we have a dual core CPU and are running 2 engines with pondering on, we are essentially trying to get 2*100% + some amount of CPU time for the GUI, out of 200% total CPU time. CPU time taken by the GUI is very significant at 1 second games.

So no matter what we do, it won't be fair, in some way.

With the "stop both clocks, then do all the GUI processing before passing the move to the opponent and starting its clock" approach, the GUI is never competing for CPU cycles with the engine that is actively thinking, and the engines get all the allocated time for active thinking.

The only unfairness in that is both engines get some extra pondering time, but I believe that's the least of the devils, since active thinking time is more important, and both engines get about the same amount of extra pondering time.

[bob]

Does your own referee program draw a graphical board and display the engine output on the screen in a scrolling window?

Nope it doesn't. It's pure command line.

I was just confirming that it's not a problem with the engine.

I had similar problem with xboard running very fast games in the past, too, and had to remove all GUI code (this is way before noGUI).

In the case of xboard, would it be possible to stop both engines' clocks while the UI updates are happening?

I doubt it actually matters with my command line referee program, but the very first thing I do when I receive a move from an engine is to stop the clock. Then I do whatever I need to do, and don't start the opponent's clock until after the move has been given to the opponent.

for the record, the noGUI option seemed to clean this up for the most part, particularly with no engine output window and such. I suspect much of it is just windows overhead however, because even with two programs, ponder=off, there are delays. Just not as much as when both programs are pondering, sending analysis to xboard, which has to scroll it in a window, while passing crap back and forth between the engines and updating the graphical board.

Peter had me looking at a couple of games he lost on ICC, and I suspect it is related to this since you still have two data streams (from/to ICC and from/to the engine) as well as the engine output window, and I'll bet he likes those "hopping knights" as well.