Stockfish, info currmove and bad UCI practice

[hgm]

Well, according to this document:

info
the engine wants to send infos to the GUI. This should be done whenever one of the info has changed.
The engine can send only selected infos and multiple infos can be send with one info command,
e.g. "info currmove e2e4 currmovenumber 1" or
"info depth 12 nodes 123456 nps 100000".
Also all infos belonging to the pv should be sent together
e.g. "info depth 2 score cp 214 time 1242 nodes 2124 nps 34928 pv e2e4 e7e5 g1f3"
I suggest to start sending "currmove", "currmovenumber", "currline" and "refutation" only after one second
to avoid too much traffic.
(...)
* currmove
currently searching this move

So, the purpose of the "info" command is merely to provide the user with information, e.g. about the move currently being searched. It is certainly not intended to provide the GUI with a list of legal moves.

Well, you can boldface the entire specs, if you want, but that won't succeed in highlighting the word 'user' anywhere. That seems to spring entirely from your imagination. I would say the info commands are there to provide the GUI with information, as the GUI is the party to receive that info.

Sure you can tell me that you don't care what the protocol author thinks you should do with the information you get. However, relying on behavior not required by the protocol is simply broken.

I dont think that washes. To do things some (most?) users want, like printing PVs, you must rely on the engine sending you those PVs. As sending PVs is not required by the protocol, in your philosophy any GUI that would print PVs should be considered 'broken', and good GUIs should never show the user any PVs. In practice, however, >99% of the users would consider a GUI that does never shows PVs broken, and an engine that doesn't send PVs to a GUI that normally shows PVs a crappy engine.

What the GUI can do for the user reflects the capabilities of the engine. If the engine is stingy with information, the user will necessarily get less. Never using information that is not mandatory sent would make it completely pointless to ever send that info, and that cannot have been the idea of adding the possibility to send it to the protocol.

The title of this thread contains the phrase 'bad UCI practice'. That is not the same as 'Non-compliance', right? It is the task of engine developers to use the protocol in such a way that their product becomes useful. And I want to draw attention to the fact that not sending the currmove infos, or delaying it to far beyond the recommendation by the specs author, damages the usefullnes of the engine.

For regular chess your problem is easy to fix: just let the GUI generate the legal moves. You can then use any info you get from the engine to improve the sorting of moves or whatever. If you don't get that information in time, then so be it.

For variants, I am sure engine authors will want to accomodate for your wishes. Just come up with a proper protocol extension that does what you need.

Actually it is better to recommendate them to stay away from UCI and use WB protocol, which solves all issues satisfactorily. It is just that there already exist quite a few UCI engines fro Xiangqi and Shogi.

I can further add that recommendations by the specs author (like any recommendations) should be understood in the context of his intentions, rather than taken as gospel. The UCI specs originally served the purpose of prescribing rules engines had to follow to be compatible with Shredder GUI. And what is good for Shredder GUI, is not necessarily good for other GUIs. So engine authors that are interested in making there engine run well on other GUIs should rather think for themselves what would be optimal behavior, rather than blindly following recommendations in the specs... As the world changes, demands in computer Chess change as well. Of course I understand that UCI is a dead protocol, carved in stone and not able to adapt to the needs of the time, but that doesn't mean that recommendations that once might have been good are still good today.

[mcostalba]

Even the advice in the specs to not send currmove/currmovenumber in the first second is not so bad, provided you send it quickly afterwards. Having to wait 1 sec before you can start excluding moves is not a big deal. Having to wait 15 sec for that, as in the case of Stockfish, is way over the limit, though.

The way I read the UCI spec, currmove is clearly not intended to inform the GUI of the legal moves. It is intended to inform the GUI, and thereby the user, of the move the engine is currently searching. Depending on this feature to extract the set of legal moves from the engine seems a rather ugly hack to me.

I fully second this, perfectly explained !

P.S: You have to wait 3 secs not 15, but the main point here, as Ronald very well wrote, is that currmove is not intended to be used as you would like....and I will not second an ugly and broken by design hack. Let me add that UCI protocol has its weakness but is well thought and especially avoids hacks and tricks, this is why I like it. The "bad practice" is you trying to force in UCI what's is not foreseen nor designed for.

[syzygy]

Well, you can boldface the entire specs, if you want, but that won't succeed in highlighting the word 'user' anywhere. That seems to spring entirely from your imagination.

I'd call it common sense.

Sure you can tell me that you don't care what the protocol author thinks you should do with the information you get. However, relying on behavior not required by the protocol is simply broken.

I dont think that washes. To do things some (most?) users want, like printing PVs, you must rely on the engine sending you those PVs. As sending PVs is not required by the protocol, in your philosophy any GUI that would print PVs should be considered 'broken', and good GUIs should never show the user any PVs.

I am saying relying on behaviour not required by the protocol. A GUI should not depend for its proper funtioning on engine behavior going beyond what the protocol requires. If it does, then that is broken. It's fine if you use engine info for improving presentation. It is broken if you need such info (not required by the spec) or the GUI won't work properly.

The title of this thread contains the phrase 'bad UCI practice'. That is not the same as 'Non-compliance', right? It is the task of engine developers to use the protocol in such a way that their product becomes useful. And I want to draw attention to the fact that not sending the currmove infos, or delaying it to far beyond the recommendation by the specs author, damages the usefullnes of the engine.

In so far as it "damages" the usefulness of the engine in the sense that the GUI now has no way of knowing what moves are legal in chess, then you are really asking developers to cater for the inadequacy of a GUI whose primary function is to be used for chess (or is this primary function again my imagination?).

Really, only very few people (me included) care about variants. What you require from variant engines is for you to decide. But a GUI should work just fine with regular chess engines that comply with the protocol.

Actually it is better to recommendate them to stay away from UCI and use WB protocol, which solves all issues satisfactorily. It is just that there already exist quite a few UCI engines fro Xiangqi and Shogi.

Then ask their authors to let their engines communicate the legal moves?

So engine authors that are interested in making there engine run well on other GUIs should rather think for themselves what would be optimal behavior, rather than blindly following recommendations in the specs...

Yes, but requesting them to communicate the set of legal moves using currmove within the first few 200ms just so that the GUI can learn what are the legal moves in chess, is just a bit odd imho. As you wrote earlier:

I think it is a mistake for engines to cater to defects in GUIs, like it is a mistake to let GUIs cater to non-compliant engines.

Btw, I'm fine with new or improved or extended protocols. I just think it's a rather ugly hack to use currmove info for determining legal moves. Maybe good enoug for variants, but not for chess.

[Uri Blass]

It is clear that stockfish has some fail low for e6-e5 at depth 2 because the score for Qg7-g3 is worse but stockfish never show a score that is worse than 1.05 for e6-e5 at depth 2

Depth 2 !?!?!?!

PV info in case of fail/high low is printed after 2 seconds of search because before is useless given that the correct resolved score is given just fractions of seconds later.

This is the actual code

Code: Select all

                // Send full PV info to GUI if we are going to leave the loop or
                // if we have a fail high/low and we are deep in the search.
                if (&#40;bestValue > alpha && bestValue < beta&#41; || SearchTime.elapsed&#40;) > 2000&#41;
                    pv_info_to_uci&#40;pos, depth, alpha, beta&#41;;

This fail high/low infor could be sueful for analysis, but you don't analyse a position with 2 seconds time. Please avoid me the embarrass to read that _now_ also the first two seconds are important to you.

It is not correct that I do not analyze position with 2 second time or less than it.

I use more time for most of the positions but there are clearly cases that I can decide based on less than 1 second that I am not interested in analyzing the specific line and go to different line

If I want to check some move that seems to me interesting and I see that after the move one side has mate score after 0.5 second of analysis then I prefer to go backward and analyze different lines.
With stockfish I need to wait 2 seconds so using it clearly waste my time relative to other engines.


I think that the suggestion that I read in another post to start sending "currmove", "currmovenumber", "currline" and "refutation" only after one second to avoid too much traffic is clearly a bad suggestion but even this suggestion is better than what stockfish does because with stockfish you need to wait 2 seconds that is worse than one second.

I think that it is clearly better to omit the if from stockfish code and make stockfish code shorter by one line and more user friendly.

I can add that it is not the only problem with stockfish and the main disadvantage of stockfish seems to be that for some reason
stockfish is the only strong engine that gives information only at the end of the iteration and I expect user friendly program to give information every time that it changes its mind.

It clearly does not happen with stockfish.
I think that stockfish need some printing of the pv after line 1025 in search.cpp

Code: Select all

if (isPvMove || value > alpha)

maybe adding the last line to the following code after line 1025 of search.cpp can be productive but I am not sure about it because I am not sure if I understand stockfish code

Code: Select all

if (isPvMove || value > alpha)
          {
              rm.score = value;
              pv_info_to_uci(pos, depth, alpha, beta);

[mcostalba]

I use more time for most of the positions but there are clearly cases that I can decide based on less than 1 second that I am not interested in analyzing the specific line and go to different line

I wouldn't call it to "analyze", I would call it to go moneky style across the keyboard. Anyhow, apart that your argumentations are quickly escalating to pure non-sense, the point is that you don't need to print fail high/low info when, a fraction of a second later, you get the full and correctly resolved PV.

With stockfish I need to wait 2 seconds so using it clearly waste my time relative to other engines.

Chess is a game of patience

maybe adding the last line to the following code after line 1025 of search.cpp can be productive but I am not sure about it because I am not sure if I understand stockfish code

Code: Select all

if (isPvMove || value > alpha)
          {
              rm.score = value;
              pv_info_to_uci(pos, depth, alpha, beta);

Wow, you read the code, and even mocked up something. I'm impressed, really.

No, it doesn't work because that code is multi thread so you cannot just print but you should take care that only one thread is printing at that time. And this requires a bit of work that, considering what is the goal to achieve, is a total waste of time and resources and would add totally useless lines of code. So I won't do.

[JuLieN]

I tried to run Stockfish under WinBoard + UCI2WB, to test how well the exclusion of moves during analysis works. It turned out not to work very well, and the reason seems to be that Stockfish is very tardy in sending the 'info currmove / currmovenumber' commands. Often it only starts sending these comands more than 3 sec after the search started. And to compound the disaster, it then does not send the complete set of moves, but starts somewhere in the middle (where it happened to be searching), so that you will have to wait for the next iteration (and until it finishes!) to get a complete set of moves. This means that the complete set of moves is often only available after 10-15 sec! (Note that you have to wait for a complete set from one iteration, and cannot coplete a partial set from a pprevious iteration with moves from the next, because engines often alter the move ordering from iteration to iteration.)

I noticed that other UCI engines often are also slow in sending currmove info, but those I tried were never that slow.

I know the UCI specs contain the recommendation to not spam the GUI during the first second, but 'info currmove / currmovenumber' cannot be considered spam. They send essential info (namely the set of legal moves) that the GUI has not seen before, and that it could need for issuing a UCI 'searchmoves' command. It is mindless repetition of the set could be considered spamming, but the first such set certainly isn't. So I would expect decent UCI engines to send at least one complete set of moves early on. It doesn't really have to be on the first iteration, but at least such that the GUI has received the info within the first second (and preferably within the first 200 msec).

Sorry to jump so late in this thread, after everything has been said already, but maybe my own testimony will add something anyway.

The first versions of my engine were always sending the currmove / currmovenumber informations, and it had two consequences:
1- some users complained that it was spamming their GUI or their shell.
2- with all OSes, console output is always relatively slow (I made no test, but the time it takes to send an output the engine has time to compute hundreds of kilo nodes.) As a result, my engine was losing some games at time control, and even when it didn't it used a big part of its time into sending outputs instead of searching.

So since a few versions my policy is:
1- send nothing before you reach a depth of 3 plies (or else you spam the GUI with hundreds of lines for nothing).
2- Starting with a depth of three plies, send something, but no more than once per second (so, yes, the engine will for instance jump from currmove 2 to currmove 20), except if it's a new pv.

Now there's no flooding anymore, and the engine never loses on time anymore.

[Don]

I tried to run Stockfish under WinBoard + UCI2WB, to test how well the exclusion of moves during analysis works. It turned out not to work very well, and the reason seems to be that Stockfish is very tardy in sending the 'info currmove / currmovenumber' commands. Often it only starts sending these comands more than 3 sec after the search started. And to compound the disaster, it then does not send the complete set of moves, but starts somewhere in the middle (where it happened to be searching), so that you will have to wait for the next iteration (and until it finishes!) to get a complete set of moves. This means that the complete set of moves is often only available after 10-15 sec! (Note that you have to wait for a complete set from one iteration, and cannot coplete a partial set from a pprevious iteration with moves from the next, because engines often alter the move ordering from iteration to iteration.)

I noticed that other UCI engines often are also slow in sending currmove info, but those I tried were never that slow.

I know the UCI specs contain the recommendation to not spam the GUI during the first second, but 'info currmove / currmovenumber' cannot be considered spam. They send essential info (namely the set of legal moves) that the GUI has not seen before, and that it could need for issuing a UCI 'searchmoves' command. It is mindless repetition of the set could be considered spamming, but the first such set certainly isn't. So I would expect decent UCI engines to send at least one complete set of moves early on. It doesn't really have to be on the first iteration, but at least such that the GUI has received the info within the first second (and preferably within the first 200 msec).

Sorry to jump so late in this thread, after everything has been said already, but maybe my own testimony will add something anyway.

The first versions of my engine were always sending the currmove / currmovenumber informations, and it had two consequences:
1- some users complained that it was spamming their GUI or their shell.
2- with all OSes, console output is always relatively slow (I made no test, but the time it takes to send an output the engine has time to compute hundreds of kilo nodes.) As a result, my engine was losing some games at time control, and even when it didn't it used a big part of its time into sending outputs instead of searching.

So since a few versions my policy is:
1- send nothing before you reach a depth of 3 plies (or else you spam the GUI with hundreds of lines for nothing).
2- Starting with a depth of three plies, send something, but no more than once per second (so, yes, the engine will for instance jump from currmove 2 to currmove 20), except if it's a new pv.

Now there's no flooding anymore, and the engine never loses on time anymore.

Although you are forced to deal with these issue's via the engine, it should be handled by the GUI. The time it takes to transfer data back and forth is negligible, even in a high level language but having the GUI processes it and then display it can be significant, depending on the hardware and libraries used to build the user interface. So to me this issue is unacceptable and can be blamed on the GUI. In essence, the GUI punishes your engine for being more complete with respect to what information is sent.

I did my own simple analysis mode tool in tcl/tk with a nice GUI, and tcl is known to be slow but I liked the idea (and I think every GUI should do this) of displaying the PV in proper SAN notation. However there is a noticeable stall while the GUI processes this information right from ply 1. I think in this case it's the conversion to SAN notation that slows things down because TK graphics are pretty snappy - it's not openGL but it's not a dog either.

This probably is solvable, it just requires proper GUI design. First of all the GUI should process stdout from the engine minimally and be prepared to stop the programs clock as soon as possible upon receiving the "bestmove xxxx" message.
The messages that come before that can be moved to a queue to be processes in turn - and if the program gets ahead of the GUI any previous info messages should be skipped. It's more work, but it's the right way to do it.

Here is an example:

The engine sends the info command "info currmove e2e4 currmovenumer 1", so the GUI displays this information.

After the GUI is finished displaying the above, the program is already on depth 3 and has sent much more information. The GUI ignores all the intermediate "info" messages and proceeds to display only the LAST message which in our example came from the 3rd iteration. This process continues and at some point the engine sends the "bestmove xxxx" message to the GUI which happens to be busy processing other strings from the engine. The GUI should IMMEDIATELY stop the programs clock, even if it is still writing information to the display. It's ok if the GUI cannot keep up with the engine as long as the clock is stopped as soon as possible after the engine moves and time is not taken displaying and processing information that is immediately going to be overwritten anyway.

There are some complexities with doing this, but it's not that difficult. It's a simpler model to simply get a message and then fully process it and move to the next message, so the process of getting the messages needs to be separated from the processing of those messages.

Don

[Uri Blass]

I use more time for most of the positions but there are clearly cases that I can decide based on less than 1 second that I am not interested in analyzing the specific line and go to different line

I wouldn't call it to "analyze", I would call it to go moneky style across the keyboard. Anyhow, apart that your argumentations are quickly escalating to pure non-sense, the point is that you don't need to print fail high/low info when, a fraction of a second later, you get the full and correctly resolved PV.

With stockfish I need to wait 2 seconds so using it clearly waste my time relative to other engines.

Chess is a game of patience

maybe adding the last line to the following code after line 1025 of search.cpp can be productive but I am not sure about it because I am not sure if I understand stockfish code

Code: Select all

if (isPvMove || value > alpha)
          {
              rm.score = value;
              pv_info_to_uci(pos, depth, alpha, beta);

Wow, you read the code, and even mocked up something. I'm impressed, really.

No, it doesn't work because that code is multi thread so you cannot just print but you should take care that only one thread is printing at that time. And this requires a bit of work that, considering what is the goal to achieve, is a total waste of time and resources and would add totally useless lines of code. So I won't do.

In other words adding code to make stockfish more user friendly is a total waste of time based on your opinion.

I guess that at least the code that I suggest can make stockfish more user friendly on one core.

[hgm]

I am saying relying on behaviour not required by the protocol. A GUI should not depend for its proper funtioning on engine behavior going beyond what the protocol requires. If it does, then that is broken. It's fine if you use engine info for improving presentation. It is broken if you need such info (not required by the spec) or the GUI won't work properly.

Well, most users would not consider it 'proper functioning' if during analysis they would not get to see any PVs or score. Yet, to display those, the GUI has to rely on the engine sending 'info pv' commands, which is not a protocol requirement.

In so far as it "damages" the usefulness of the engine in the sense that the GUI now has no way of knowing what moves are legal in chess, then you are really asking developers to cater for the inadequacy of a GUI whose primary function is to be used for chess (or is this primary function again my imagination?).

Indeed, that is your imagination. UCI2WB its prime design goal is to allow Xiangqi and Shogi engines to run under WinBoard, but it would be a nice bonus if people that primarily play these games, and thus have configured WinBoard accordingly, could occasionally run Chess engines without to much loss of functionality. But that is a bit beyond the scope of this thread.

Yes, but requesting them to communicate the set of legal moves using currmove within the first few 200ms just so that the GUI can learn what are the legal moves in chess, is just a bit odd imho.

Well, I would be already happy if engine authors followed the 'official' recommendation to start sending it after 1 sec. But some people apparently take pride in beefing up the reputation of their engine as 'unfriendly for analysis'...

[syzygy]

In so far as it "damages" the usefulness of the engine in the sense that the GUI now has no way of knowing what moves are legal in chess, then you are really asking developers to cater for the inadequacy of a GUI whose primary function is to be used for chess (or is this primary function again my imagination?).

Indeed, that is your imagination.

Ok, if xboard and winboard are not anymore about chess, then I give up. In the past they were, though.