Some more WB protocol extensions

[bob]

I know a lot has been discussed about this in the past, but I want to re-open the discussion, as I might actually implement some new stuff. This applies to the following isues:

1. Multi-PV

This seems the least controversial one: simply allow an engine to send feature multipv=1 at startup. If the GUI has received this, it can then send multipv N to the engine, to request it to calculate and send (as Thinking Output) the best N lines. The engine can decide how many lines it will actually send; the command is a request only. (After all, there might be only one legal move...)

2. Hash clearing

The specs on the memory command will be beefed up a bit, to imply clearing the hash table (and other tables) even if the requested size is equal to the current size. Currently WinBoard sends the memory command after new on every game, but future interfaces could refrain from that, if the hash size can stay the same and they don't really want to reset the engine to a known state. To the specs of the new command the recommendation will be added that it should leave the hash table alone.

3. Thinking Output

Currently there is a hideous feature in WB protocol called 'periodic updates', whereby the engine is polled by the GUI for information it might or might not have. I want to deprecate that, as it is a clear case of totally wrong design. Polling is awful; the engine should volunteer this information, as it is the engine that knows when it has new information to present. And logically, the information belongs with the Thinking Output. In fact it even duplicates part of it (depth, time, nodes).

So I plan to make an extension to the specs for Thinking-Output lines, which can include this information. The engine can then send this info not only during analysis, but also during pondering and regular search. And as the PV in the olds protocol specs is a free-format field, there is automatic backward compatibility. Old interfaces that do not know about the new standard, would simply display the info as part of the PV. In fact, I already know some engines that use this technique (e.g. NanoSzachy), and it looks quite nice.

In particular, I want the new format to look as

depth score time nodes tbhits {move movenr/totalmoves} PV +-

In front of the PV, between braces, the engine can display the move it is currently considering in the root, the sequence number of that move, and the total number of root moves. (I.e., the info that formerly was in the stat01 lines.) In front of it, there can be an optional number to indicate the number of tablebase hits. Perhaps some more optional fields could be squeezed in here in the future; I would like to keep their number low, though, in order not to push the PV out of view on old interfaces. Prepending 0 {f3e5 3/20} (14-15 characters) seems still acceptable.

The 'time' parameter should specify CPU-time used (sum of all threads for SMP engines), so that the GUI can calculate CPU load. Sending wall-clock time here is kind of redundant, as the GUI can measure this itself.

For the record, in the world of non-windows, which is a sizable world, CPU time is problematic. There is no standard on obtaining this information. Some systems return total time for all threads, some return the time for the specific thread making the request. In Crafty, I am unable to report this time as I removed that code completely. I can report elapsed time only, as that is all I have. I don't particularly like the idea of CPU time anyway. A parallel program can use less than N*elapsed_time seconds (where it is using N threads) for several reasons. Other threads are preempting the chess engine. The chess engine is doing EGTB accesses that block for I/O and consume no CPU, the parallel search uses blocking for whatever reason, such as thread synchronization, MUTEX locks, etc. The "cpu load" doesn't really give a lot of information by itself.

Mate scores should be standardized, and preferably encoded in a way that does not break old interfaces. In Joker I use the system of encoding a mate in N (moves!) in the Thinking Output as 100000+N, and mated in N as -(100000+N). This keeps it purely numeric, and in a way that makes you immediately read the DTM on old interfaces, rather than having to do a mental subtraction from an obscure number such as 32,768. GUIs aware of the standard would of course convert it to a text like MATE6 in the score field.

At the end of the PV there can be an optional + or - to indicate fail high or fail low. If I understand correctly, the PV will never be longer than 1 move in that case, so it will always be in view.

Probably an unsafe assumption. To break this, one needs to (a) protect the best move when you store an entry that is UPPER (fail low, no best move) by keeping the best move for this position from the position you are replacing (assuming the two positions are the same of course) and then (b) doing hash probes to fetch the PV to display it. That would allow a longer PV.

4. Multi-Session Time Control

When an engine sends feature xlevel=1 at startup, the GUI will send it a multi-session level command at the beginning of every session.
This is just like the old-style level command, except that it has an arbitrary number of MPS, TC, INC triplets on the same line. The first of these triplets will contain the parameters for the upcoming session. The following parameters will offer the engine a preview of what to expect in subsequent sessions.

Trustful engines could look only at the first three parameters, to know what to expect for the next MPS moves. (The TC will actually be a bit redundant, as it will immediately be superseeded by the following time command.) They will naively fall, however, for "treacherous" time controls like 40/60+100/0:01 (specified by level 40 60 0 100 0:01 0 before move 1 and level 100 0:01 0 before move 41). If the engine wants to be more clever, it can always look ahead, as far into the future as it deems necessary. (I can imagine that after seeing 40/60+20/15+..., it is not terribly interesting to know what will happen in the third session before the second session starts.)

There will be no guarantee that what the engine is told by the GUI is true; the engine should always obey the last level command that is received, even if that contradicts to what was said before. It would be perfectly valid for GUIs to implement a sort of handicap TC mode where the engine suddenly and unexpectedly has to speed up after a certain move, even before the end of he current session. So after receiving level 40 10 0 at the start of the game, promising classical 40/10 repeating TC, the GUI could surprise the engine by a level 0 5 0 command at move 30, forcing the engine to finish the game in 5 min under sudden-death rules. Conducting such games could by useful for certain types of testing, so the protocol should support this possibility.

FEEDBACK WELCOME!

[Michel]

There are really two pieces of information - one of which, yes, the engine knows when it changes but one of which is useful to poll for. To get the current number of nodes, I think the GUI should continue to poll - even if the protocol allows the engine to send when the current move it's thinking about changes.

I tend to agree to something like that as well. One thing I hate about UCI engine is that they spam the console with node updates. It seems much more logical that the GUI would ask the engine every once in a while (e.g. every second) how many nodes it has searched so far.

[michiguel]

3. Thinking Output

Currently there is a hideous feature in WB protocol called 'periodic updates', whereby the engine is polled by the GUI for information it might or might not have.

There are really two pieces of information - one of which, yes, the engine knows when it changes but one of which is useful to poll for. To get the current number of nodes, I think the GUI should continue to poll - even if the protocol allows the engine to send when the current move it's thinking about changes.

Andy.

I agree. If you want to know how the nodes are progressing, and most importantly, what move is being considered, you have to poll.

Miguel

[bob]

I was simply objecting to the use of the word "hideous" for a command that allows the GUI to ask information from the engine.

There is no need to use such charged words when discussing a communication protocol.

I agree with HGM. Polling is a poor way of doing this. So rather than "hideous" how about "poor approach" or "crappy approach" or "antiquated approach" or "sophmoric approach" or something equally descriptive. Polling is never a good design. It wasn't a good design in MSDos when it polled a disk drive over and over to ask "are you done, yet?". It is not really a good design today when we ask "is there a move to read". We do this because there is no compatible way in windows to avoid this. In unix, it is trivial to use asynchio. One can always eliminate the polling by using a second thread, and then some sort of signaling mechanism to tell the search when you have a new move, etc. And that is a better design as well. In short, polling is a poor solution to a simple problem.

If you want to see something every N seconds, then we should define a mechanism to make that happen _automatically_. Crafty has a "burp" parameter that tells it how often to update the current display so that we can tell it is alive in tournament play when ICGA rules do not allow you to hit a key to ask "are you alive". So a solution would be to either (a) let the GUI tell the engine how often to provide some sort of update or (b) let the user tell the engine how frequently he wants to see information (about all that this shows is total nodes and current move being searched, most engines will tell the gui when the score or PV changes or there is a fail-high or fail-low automatically. I think the entire idea could be dropped and just let the engine tell the GUI about current state whenever it feels like it, or not at all.

[bob]

3. Thinking Output

Currently there is a hideous feature in WB protocol called 'periodic updates', whereby the engine is polled by the GUI for information it might or might not have.

There are really two pieces of information - one of which, yes, the engine knows when it changes but one of which is useful to poll for. To get the current number of nodes, I think the GUI should continue to poll - even if the protocol allows the engine to send when the current move it's thinking about changes.

Andy.

I agree. If you want to know how the nodes are progressing, and most importantly, what move is being considered, you have to poll.

Miguel

Why? Crafty spits out an update every 15 seconds just to let me know what is going on and that it is still alive. It doesn't have to be polled to supply this.

[Zach Wegner]

2. Hash clearing

The specs on the memory command will be beefed up a bit, to imply clearing the hash table (and other tables) even if the requested size is equal to the current size. Currently WinBoard sends the memory command after new on every game, but future interfaces could refrain from that, if the hash size can stay the same and they don't really want to reset the engine to a known state. To the specs of the new command the recommendation will be added that it should leave the hash table alone.

I too would prefer a new command here ("reset" sounds good). Allowing the user to clear the hash is pretty useful. "new" has its own semantics, starting games. Some engines might want to clear the hash, others not. The standard should be to keep it, but only if there is a reset command. Without a reset, the default should be to clear on "new". "memory" should only be sent when memory changes, "new" only for a new game, and "reset" only to clear hash. Doing otherwise would really just cause confusion here.

3. Thinking Output

Currently there is a hideous feature in WB protocol called 'periodic updates', whereby the engine is polled by the GUI for information it might or might not have. I want to deprecate that, as it is a clear case of totally wrong design. Polling is awful; the engine should volunteer this information, as it is the engine that knows when it has new information to present. And logically, the information belongs with the Thinking Output. In fact it even duplicates part of it (depth, time, nodes).

So I plan to make an extension to the specs for Thinking-Output lines, which can include this information. The engine can then send this info not only during analysis, but also during pondering and regular search. And as the PV in the olds protocol specs is a free-format field, there is automatic backward compatibility. Old interfaces that do not know about the new standard, would simply display the info as part of the PV. In fact, I already know some engines that use this technique (e.g. NanoSzachy), and it looks quite nice.

In particular, I want the new format to look as

depth score time nodes tbhits {move movenr/totalmoves} PV +-

In front of the PV, between braces, the engine can display the move it is currently considering in the root, the sequence number of that move, and the total number of root moves. (I.e., the info that formerly was in the stat01 lines.) In front of it, there can be an optional number to indicate the number of tablebase hits. Perhaps some more optional fields could be squeezed in here in the future; I would like to keep their number low, though, in order not to push the PV out of view on old interfaces. Prepending 0 {f3e5 3/20} (14-15 characters) seems still acceptable.

The 'time' parameter should specify CPU-time used (sum of all threads for SMP engines), so that the GUI can calculate CPU load. Sending wall-clock time here is kind of redundant, as the GUI can measure this itself.

Mate scores should be standardized, and preferably encoded in a way that does not break old interfaces. In Joker I use the system of encoding a mate in N (moves!) in the Thinking Output as 100000+N, and mated in N as -(100000+N). This keeps it purely numeric, and in a way that makes you immediately read the DTM on old interfaces, rather than having to do a mental subtraction from an obscure number such as 32,768. GUIs aware of the standard would of course convert it to a text like MATE6 in the score field.

At the end of the PV there can be an optional + or - to indicate fail high or fail low. If I understand correctly, the PV will never be longer than 1 move in that case, so it will always be in view.

I think if we're going to standardize thinking output, we should do it properly. This is one area where UCI excels. Though the syntax is ugly, the standardization is very nice. Xboard has a problem here due to backward compatibility. I think in this case, abandoning backward compatibility with old GUIs would be worth it for better thinking output.

I think some more flexibility would be desirable here, if we're going to redesign it. I have researched algorithms that don't use centipawns or depth, so requiring them in the thinking output is bad IMO. Since engines might have more resolution to their scores, floating point would be a good idea. Adding the possibility to specify scores as probabilities or mate-in-N values too. Scores should always be from the engine's POV. PVs should be in a standard format (coordinate or SAN I suppose), so that the GUI can interpret it. Sending time as CPU-time only is a good idea. Since the protocol now supports an arbitrary number of TB types, sending the type of TB hits would be good too.

My proposal would be something like this:

Code: Select all

d=14 sd=26 --> depth, seldepth
t=1:02.35 --> time, self explanatory
mv={f1b5} mv={f1b5 3/30} --> current move, move number (optional)
s=+0.4832 s=+#47 s=57.4% --> score in different units, pawns, mate, and probability respectively
pv={e2e4 c7c5 g1f3}  -->  pv, self explanatory
tb[nalimov]=4212 --> nalimov tb hits

The engine could send as much or as little as desired, and should just end it with a newline in order to flush the line to the GUI.

4. Multi-Session Time Control

When an engine sends feature xlevel=1 at startup, the GUI will send it a multi-session level command at the beginning of every session.
This is just like the old-style level command, except that it has an arbitrary number of MPS, TC, INC triplets on the same line. The first of these triplets will contain the parameters for the upcoming session. The following parameters will offer the engine a preview of what to expect in subsequent sessions.

Trustful engines could look only at the first three parameters, to know what to expect for the next MPS moves. (The TC will actually be a bit redundant, as it will immediately be superseeded by the following time command.) They will naively fall, however, for "treacherous" time controls like 40/60+100/0:01 (specified by level 40 60 0 100 0:01 0 before move 1 and level 100 0:01 0 before move 41). If the engine wants to be more clever, it can always look ahead, as far into the future as it deems necessary. (I can imagine that after seeing 40/60+20/15+..., it is not terribly interesting to know what will happen in the third session before the second session starts.)

There will be no guarantee that what the engine is told by the GUI is true; the engine should always obey the last level command that is received, even if that contradicts to what was said before. It would be perfectly valid for GUIs to implement a sort of handicap TC mode where the engine suddenly and unexpectedly has to speed up after a certain move, even before the end of he current session. So after receiving level 40 10 0 at the start of the game, promising classical 40/10 repeating TC, the GUI could surprise the engine by a level 0 5 0 command at move 30, forcing the engine to finish the game in 5 min under sudden-death rules. Conducting such games could by useful for certain types of testing, so the protocol should support this possibility.

I think sending level in the middle of a game is just going to cause bugs, even if the behavior is well defined. I don't see it as being too useful anyways.

Also, using only two parameters per session, except the last might be simpler. Or perhaps a delimiter between sessions (semicolon, plus, something). I don't feel too strongly about these points though, since I don't plan on implementing multi-session TCs.

[Michel]

agree with HGM. Polling is a poor way of doing this. So rather than "hideous" how about "poor approach" or "crappy approach" or "antiquated approach" or "sophmoric approach" or something equally descriptive. Polling is never a good design.

Polling for something to happen is always bad.

Polling to measure the progress of something is not necessarily bad.

Would you like your wall clock to emit a beep every second or do you prefer to look at it when you need to know the time?

[michiguel]

Would it not be better to have another command "clearbrain" or "clear" (or whatever name) to separate those tasks rather than interfering with previous commands? In fact, "clearing" is a different conceptual task that may be useful even if the engine is not using hashtables (for instance, clearing killers, or any variable inherited from the previous search).

The command reset has also been proposed for this, but that was before the memory command existed. When memory forces you to re-size the hash table, I guess there is no practical way to do it without clearing it. We might as well make it a recommendation to always do that.

Not necessarily. When I receive memory, hash is cleared by force but not killers and history. When I mentioned clearing the "brain", it is a more broad concept that goes beyond "hash".

I have considered adding a new command for this, but it seems a waste to add a new command for one that does already do nearly the same. The fewer commands the better. Especially when at the moment there is no practical use for this command. In theory it could be used for epd-suite testing, but that is not something that WinBoard supports.

By the way, there is and epd2wb utility written by B. Moreland and modified by T. Mayer that does that.

The more different commands you make, the less likely engines will be to implement any of them. On the other hand, if a command can be usful in many situations, it enhances the chances that people wil want to implment it. Of course we could put it in the specs as merely a recommendation: bring the engine in as well-defined a state as you ever care it to be in. People wil implement what they want anyway, and if they do not care about epd suites, why shoud they bother with clearing killer tables etc. That 'new' should not clear the hash is also only a recommendation. Some engne authors prefer clearing the hash before every move (for reproducibility of errors), and it would be logical for them to aso clear on 'new'. They should have that freedom. Some engines might not even have a hash table...

That is why I think reset or clear will be useful when the user really want to start from zero.

Miguel

[hgm]

For the record, in the world of non-windows, which is a sizable world, CPU time is problematic. There is no standard on obtaining this information. Some systems return total time for all threads, some return the time for the specific thread making the request. In Crafty, I am unable to report this time as I removed that code completely. I can report elapsed time only, as that is all I have. I don't particularly like the idea of CPU time anyway. A parallel program can use less than N*elapsed_time seconds (where it is using N threads) for several reasons. Other threads are preempting the chess engine. The chess engine is doing EGTB accesses that block for I/O and consume no CPU, the parallel search uses blocking for whatever reason, such as thread synchronization, MUTEX locks, etc. The "cpu load" doesn't really give a lot of information by itself.

The reason I proposed this is that there was demand for having the engine report its CPU load. All the difficulties you mention would also apply if the engine would have to calculate this load by itself by dividing CPU-time by wall-clock time. So perhaps it is good to split up the debate into two sub-issues:

1) would it be better to have the engine calculate its CPU load and sent it to the GUI as a separate parameter, or should it simply send the raw CPU time, and let the GUI do the math? I would aim for the latter.

2) Is any of this feasible? That you took out the code, is not the same as it being impossible... If it is not feasible under non-Windows OS, my take would be then just to "cheat" and report wall-clock time (or wallClockTime * nrOfThreads), so that a GUI would report 100% CPU usage. Not much harm done, I would say. If some engines _can_ meaningfully report this quantity in _some_ situations, the protocol should provide a way to do it. Engines that for som reason cannot or don't want to comply, can simply sent some default. There are plenty of WinBoard engines that always send 0 for the nod count, for instance.

Probably an unsafe assumption. To break this, one needs to (a) protect the best move when you store an entry that is UPPER (fail low, no best move) by keeping the best move for this position from the position you are replacing (assuming the two positions are the same of course) and then (b) doing hash probes to fetch the PV to display it. That would allow a longer PV.

OK, so you are explaing us how engines could foul it up. But it takes a conscious action of the programmer to make the engine emit the + or - fail indications, and if he takes care to program that, he might as well put in code to suppress the PV. My point was that a long PV is not _needed_ in these situations, so there is no downside to requiring that these symbols go behind the PV. You know if you failed high, and can then print the failing move, suffixed with a + (or ++, or whatever we settle on). If you fail low, you can just print a -, and no move.

[michiguel]

I was simply objecting to the use of the word "hideous" for a command that allows the GUI to ask information from the engine.

There is no need to use such charged words when discussing a communication protocol.

I agree with HGM. Polling is a poor way of doing this. So rather than "hideous" how about "poor approach" or "crappy approach" or "antiquated approach" or "sophmoric approach" or something equally descriptive. Polling is never a good design. It wasn't a good design in MSDos when it polled a disk drive over and over to ask "are you done, yet?". It is not really a good design today when we ask "is there a move to read". We do this because there is no compatible way in windows to avoid this. In unix, it is trivial to use asynchio. One can always eliminate the polling by using a second thread, and then some sort of signaling mechanism to tell the search when you have a new move, etc. And that is a better design as well. In short, polling is a poor solution to a simple problem.

I have a second thread that deals with the "polls". That is still a "poll" from the GUI. I do not know then what you mean by poll because it is not eliminated.

If you want to see something every N seconds, then we should define a mechanism to make that happen _automatically_.

The user may want to know only once what is going on and do it manually, so being spammed every N seconds just in case the user wants to know something is hideous

Crafty has a "burp" parameter that tells it how often to update the current display so that we can tell it is alive in tournament play when ICGA rules do not allow you to hit a key to ask "are you alive".

That is fine, but ICGA tournaments are a small fraction of the universe of CC. I think you agree

So a solution would be to either (a) let the GUI tell the engine how often to provide some sort of update or (b) let the user tell the engine how frequently he wants to see information (about all that this shows is total nodes and current move being searched, most engines will tell the gui when the score or PV changes or there is a fail-high or fail-low automatically. I think the entire idea could be dropped and just let the engine tell the GUI about current state whenever it feels like it, or not at all.

If the engine is capable to relay information when it is asked, I do not see the problem.

Miguel