Stockfish, info currmove and bad UCI practice

[hgm]

Yes I'm aware of UCI_AnalyseMode. But I think not all GUIs support it. No big deal with go infinite i think. Should work everywhere.

I think it is a mistake for engines to cater to defects in GUIs, like it is a mistake to let GUIs cater to non-compliant engines. It will decrease the chances that the defects will ever be repaired, and will lead to a viscious cycle of degeneration of the standards.

If an engine should behave differently in analysis than in game play, I would advice to do that based on UCI_AnalyseMode, like the specs prescribe.

My engine supports both searchmoves and multipv but unfortunately it doesn't report currmove when infinite search is issued. I wanted to minimize traffic that way. I was to fix that in the next version but atm i have nothing to release

Well, limiting bandwidth is not the primary concern in analysis mode. Even the advice in the specs to not send currmove/currmovenumber in the first second is not so bad, provided you send it quickly afterwards. Having to wait 1 sec before you can start excluding moves is not a big deal. Having to wait 15 sec for that, as in the case of Stockfish, is way over the limit, though.

Regarding standards, I saw a GUI (Arena) happily letting an engine play matches in multipv mode. Nothing wrong with that, sure. But that certainly was not intended because searchmoves and multipv are only meaningful is analysis mode.

Well, having a TC mode in human-engine games where the engine just thinks (using 'go infinite') until the user forces it to move, does not seem a completely ridiculous thing to me. But this makes it dangerous to assume that 'go infinite' always means analysis. As, apparently, the designer of UCI protocol agreed.

[michiguel]

I tried to run Stockfish under WinBoard + UCI2WB, to test how well the exclusion of moves during analysis works. It turned out not to work very well, and the reason seems to be that Stockfish is very tardy in sending the 'info currmove / currmovenumber' commands. Often it only starts sending these comands more than 3 sec after the search started. And to compound the disaster, it then does not send the complete set of moves, but starts somewhere in the middle (where it happened to be searching), so that you will have to wait for the next iteration (and until it finishes!) to get a complete set of moves. This means that the complete set of moves is often only available after 10-15 sec! (Note that you have to wait for a complete set from one iteration, and cannot coplete a partial set from a pprevious iteration with moves from the next, because engines often alter the move ordering from iteration to iteration.)

I noticed that other UCI engines often are also slow in sending currmove info, but those I tried were never that slow.

I know the UCI specs contain the recommendation to not spam the GUI during the first second, but 'info currmove / currmovenumber' cannot be considered spam. They send essential info (namely the set of legal moves) that the GUI has not seen before, and that it could need for issuing a UCI 'searchmoves' command. It is mindless repetition of the set could be considered spamming, but the first such set certainly isn't. So I would expect decent UCI engines to send at least one complete set of moves early on. It doesn't really have to be on the first iteration, but at least such that the GUI has received the info within the first second (and preferably within the first 200 msec).

I believe a GUI should have it's own move generator and not assume anything. If an engine plays an illegal move, how do you adjudicate the win? If any engine says it has won when it hasn't, what can you do?

I can understand however that for a GUI that must easily be able to support a large variety of games, it might be more feasible to rely on some help from the engines.

An ideal protocol may have a command GUI-->engine that says "get_legals" and the engine should reply

legals a2a3 a3a4 b2b3 b2b4 etc. etc.

That would be useful for many reasons and solve many problems.

In HGM's particular situation, I think the adaptor should be modified to translate new winboard's exclude <move> to searchmoves <moves>. The adaptor should have a move generator then.

Miguel

[hgm]

An ideal protocol may have a command GUI-->engine that says "get_legals" and the engine should reply

legals a2a3 a3a4 b2b3 b2b4 etc. etc.

That would be useful for many reasons and solve many problems.

The WB 'bk' command sort of does this. In fact, Polyglot 2.0.1 was modified to send the full list of legal moves in reply to the 'bk' command in analysis mode. The idea was that there could be much information in the order of the moves. An engine receiving the 'bk' command could for instance do an open-window search for all root moves, and use the scores to sort the moves (and actually report the scores with the moves) so the GUI could use this information to make a logical presentation of the moves to the user for excluding them from the search.

In Polyglot this is pretty useless, as it can't do any more than the GUI can with its own legal-move generator. Polyglot is no engine, and cannot assign scores. So it would be dependent on collecting this info from the engine. E.g. from the order of the moves in the info currmove commands.

In HGM's particular situation, I think the adaptor should be modified to translate new winboard's exclude <move> to searchmoves <moves>. The adaptor should have a move generator then.

Well, for UCI2WB that is not acceptable, as it is supposed to be a cross-variant adapter, completely free of rule knowledge. Nice as this solution might be for the variant you write the move generator for, it makes the adapter immediately completely unusable for any other variant. UCI2WB can be used to run Chess, Xiangqi, Shogi and S-Chess engines. The only requirement is they are UCI. (Or USI, in case of Shogi.) Polyglot has gone that route, and as fine a piece of software it is for use in Chess, it is completely useless for running Xiangqi UCI engines.

And a move generator still cannot order the moves by relevance. Of course I can let the adapter run the engine in multi-PV mode for a few hundred msec when cnfronted with a new position before starting the real analysis, requesting 50 PVs, and get the info from the PVs. But not all engines support multi-PV, and even those that do can set a limit to the number of PVs. So that is not really a general solution either.

[syzygy]

Even the advice in the specs to not send currmove/currmovenumber in the first second is not so bad, provided you send it quickly afterwards. Having to wait 1 sec before you can start excluding moves is not a big deal. Having to wait 15 sec for that, as in the case of Stockfish, is way over the limit, though.

The way I read the UCI spec, currmove is clearly not intended to inform the GUI of the legal moves. It is intended to inform the GUI, and thereby the user, of the move the engine is currently searching. Depending on this feature to extract the set of legal moves from the engine seems a rather ugly hack to me.

I agree that for variants it is reasonable to ask engines for help in determining the set of legal moves, but imho a chess GUI should know the rules of chess (the real game) and be able to generate the set of legal moves. I don't see why regular chess engines should need to adjust for problems created by variants (and note that my engines plays variants). That also enables the GUI to convert ugly "f1g3" notation into much easier for humans to parse "Nf3" notation. (At least the UCI engines I have seen seem to produce that type of output, I don't know how xboard/winboard currently handles this.)

[Uri Blass]

Of course stockfish is open source so it should not be hard to make stockfish user friendly like other engines by changing the code but it seems that we need new authors who are interested to work on the code of this program to make it user friendly(I expect nothing from the current authors after they refused to fix the fact that stockfish does not show information about its fail lows).

Stockfish is not only open source, it is publicly hosted on Github. If you were willing to propose a patch to help analysis mode, I'm sure it would be accepted!

What is the fail-low issue that you are describing?

It is something that has been fixed months ago and is fixed already in 2.2.2, but please don't remove from people the subtle pleasure to say cheap bul...ts

I do not know if it is fixed(the fact that stockfish did not give information about fail low) but
I understood from the response that I got at that time that the developers do not plan to fix it and only plan to give less information and practically I see less information.

If it does not cost me too much time I will try to find the relevant post of me in this forum about it.

[hgm]

The specs say nothing about for which purpose commands are intended. Which is just as well, as this is not the task of protocol specs. These should just specify what engines and GUIs have to do. It is then up to the designer how he wants to use this information.

It is true that the protocol does not put a strict requirement on sending any of the defined infos. That holds for currmove as well as pv info. In principle only the 'bestmove' command is required, and the engine could be totally silent otherwise. But it is also clear that such an engine would be pretty useless for most applications; even engine-engine games would be pretty uninteresting without annotation by depth / score info.

So if authors want their engine to be useful, they should not be stingy with output. The currmove output can of interest for several applications, and it would therefore be unwise to unconditionally withhold it. It could be made subject to an option setting, of course, but even that is not recommended in the specs. (While for some other infos, it is specified as mandatory!)

I can add that it is a bit inconsistent to worry about the amount of traffic when you use UCI. If you are worried about traffic, you should use WinBoard protocol. That saves you a multiple of what you could save by leaving out some info currmoves. (Say a factor 50, rather than a factor 2...)

[Uri Blass]

I do not find my relevant post but I find that the fail low bug is not fixed
unlike what marco claims(so maybe he means to something else).

Here is an example from my last tournament game and I used free arena3
and free stockfish2.2.2 with 64 mbytes hash and gave stockfish the position without the game so probably everybody can reproduce it.

It is clear that stockfish has some fail low for e6-e5 at depth 2 because the score for Qg7-g3 is worse but stockfish never show a score that is worse than 1.05 for e6-e5 at depth 2

In other words stockfish hide from the user information about the fail low
and generally it also hide from the user information about the score of the first move(see the behaviour at depth 13 when stockfish probably has a real score and pv for the move Qg7-h6 that is the best move at depth 12 and the first move at depth 13 but does not show the real score to the user).

I find it clearly a bad behaviour and I expect user friendly engines to show score and pv always not only at the end of the iteration but also at the time that the program found the relevant score and pv(if the engine does not like to show score and pv twice then the best is to show it at the time that the computer found them and not later that means that the user need to wait to the end of the iteration to see the pv)

[d]4rr1k/3b2qP/p3pp2/1p1p4/5P1P/3B4/PPP1Q3/2K1R2R b - - 0 1

Stockfish-222-32-ja:
1/1 00:00 60 759 -1.05 e6-e5
2/3 00:00 862 10,911 -1.49 Qg7-g3 Re1-g1
3/4 00:00 1,185 15,000 -1.61 Qg7-g3 Re1-g1 Qg3xf4+ Kc1-b1
4/7 00:00 2,222 28,126 -1.45 Qg7-g3 Qe2-h2 Qg3xh2 Rh1xh2
5/7 00:00 5,264 66,632 -1.41 Qg7-h6 Kc1-b1 f6-f5 Rh1-g1 Kh8xh7
6/12 00:00 21,999 175,992 -0.96 Qg7-h6 Qe2-f3 f6-f5 Qf3-g3 Re8-c8 Rh1-g1 Qh6xh7
7/12 00:00 26,524 212,192 -0.92 Qg7-h6 Qe2-f3 f6-f5 Qf3-g3 Qh6xh7 Rh1-g1 Rf8-g8
8/15 00:00 60,296 320,723 -1.29 Qg7-h6 Rh1-g1 e6-e5 Bd3-g6 Qh6xf4+ Kc1-b1 Re8-e7 Qe2-g2 Bd7-e6 Bg6-f7 Qf4-g4 Qg2xg4 Be6xg4 Bf7xd5
9/18 00:00 77,983 356,086 -1.29 Qg7-h6 Rh1-g1 e6-e5 Bd3-g6 Qh6xf4+ Kc1-b1 Re8-e7 Qe2-g2 Bd7-e6 Bg6-f7 Qf4-g4 Qg2xg4 Be6xg4 Bf7xd5 Re7xh7 Rg1xg4
10/18 00:00 107,535 381,329 -1.29 Qg7-h6 Rh1-g1 e6-e5 Bd3-g6 Qh6xf4+ Kc1-b1 Re8-e7 Qe2-g2 Bd7-e6 Bg6-f7 Qf4-g4 Qg2xg4 Be6xg4 Bf7xd5 f6-f5
11/18 00:00 144,813 420,968 -1.09 Qg7-h6 Rh1-g1 e6-e5 Bd3-g6 Qh6xf4+ Kc1-b1 Re8-e7 Qe2-g2 Bd7-e6 Bg6-f7 Qf4-g4 Qg2-f1 Qg4xg1 Qf1xg1 Rf8xf7 Qg1-g8+
12/22 00:00 317,224 461,081 -1.33 Qg7-h6 Rh1-g1 e6-e5 Bd3-g6 Qh6xf4+ Kc1-b1 Re8-e7 Qe2-g2 Bd7-e6 Bg6-f7 Be6-g4 Re1-f1 Qf4-e4 Bf7xd5 Qe4xg2 Rg1xg2 Re7-g7 Bd5-e4
13/29 00:02 948,146 477,655 -1.05 e6-e5 Rh1-g1 Qg7-h6 Bd3-g6 Qh6xf4+ Kc1-b1 Re8-e7 Qe2-g2 Bd7-e6 h4-h5 Qf4-h6 Bg6-f5 Qh6-g5 Qg2xg5 f6xg5 Bf5xe6 Re7xe6 Rg1xg5 e5-e4 Rg5xd5 Kh8xh7

[Uri Blass]

Note that Stockfish2.0.1 was clearly more user friendly then later versions.

New game - Stockfish 2.2.2 JA, 5'/40+5'/40+5'/40
[d]4rr1k/3b2qP/p3pp2/1p1p4/5P1P/3B4/PPP1Q3/2K1R2R b - - 0 1

Analysis by Stockfish 2.0.1 JA:

24...e6-e5
± (0.88) Depth: 1/1 00:00:00
24...e6-e5 25.Re1-g1
± (1.33) Depth: 2/2 00:00:00
24...Qg7-g3 25.Re1-g1
± (1.29) Depth: 2/2 00:00:00
24...Qg7-g3 25.Re1-g1 Qg3xf4+ 26.Kc1-b1
± (1.37) Depth: 3/4 00:00:00
24...Qg7-g3 25.Rh1-g1 Qg3xf4+ 26.Kc1-b1 f6-f5
± (0.88) Depth: 4/5 00:00:01
24...Qg7-g3 25.Rh1-g1 Qg3xf4+ 26.Kc1-b1 f6-f5 27.Qe2-g2
+- (1.89) Depth: 5/6 00:00:01
24...Qg7-f7 25.Rh1-g1 f6-f5 26.a2-a4 Kh8xh7 27.a4xb5 a6xb5 28.Bd3xb5 Bd7xb5 29.Qe2xb5
+- (1.69) Depth: 5/10 00:00:01
24...Qg7-h6 25.Rh1-g1 e6-e5 26.Qe2-g2 e5-e4
± (1.25) Depth: 5/10 00:00:01


See the relatively nice behaviour of stockfish at depth 2 and 5.
It shows the fail low at depth 2 for e5 unlike Stockfish2.2.2

At depth 5 it shows Qg3 with some pv and later Qf7 with some pv and later Qh6 with some pv so the user can know better why it changed it's mind.

Later versions seem to show pv only for one move at every iteration
so the user get less information.

[syzygy]

The specs say nothing about for which purpose commands are intended.

Well, according to this document:

info
the engine wants to send infos to the GUI. This should be done whenever one of the info has changed.
The engine can send only selected infos and multiple infos can be send with one info command,
e.g. "info currmove e2e4 currmovenumber 1" or
"info depth 12 nodes 123456 nps 100000".
Also all infos belonging to the pv should be sent together
e.g. "info depth 2 score cp 214 time 1242 nodes 2124 nps 34928 pv e2e4 e7e5 g1f3"
I suggest to start sending "currmove", "currmovenumber", "currline" and "refutation" only after one second
to avoid too much traffic.
(...)
* currmove
currently searching this move

So, the purpose of the "info" command is merely to provide the user with information, e.g. about the move currently being searched. It is certainly not intended to provide the GUI with a list of legal moves.

Sure you can tell me that you don't care what the protocol author thinks you should do with the information you get. However, relying on behavior not required by the protocol is simply broken.

For regular chess your problem is easy to fix: just let the GUI generate the legal moves. You can then use any info you get from the engine to improve the sorting of moves or whatever. If you don't get that information in time, then so be it.

For variants, I am sure engine authors will want to accomodate for your wishes. Just come up with a proper protocol extension that does what you need.

[mcostalba]

It is clear that stockfish has some fail low for e6-e5 at depth 2 because the score for Qg7-g3 is worse but stockfish never show a score that is worse than 1.05 for e6-e5 at depth 2

Depth 2 !?!?!?!

PV info in case of fail/high low is printed after 2 seconds of search because before is useless given that the correct resolved score is given just fractions of seconds later.

This is the actual code

Code: Select all

                // Send full PV info to GUI if we are going to leave the loop or
                // if we have a fail high/low and we are deep in the search.
                if ((bestValue > alpha && bestValue < beta) || SearchTime.elapsed() > 2000)
                    pv_info_to_uci(pos, depth, alpha, beta);

This fail high/low infor could be sueful for analysis, but you don't analyse a position with 2 seconds time. Please avoid me the embarrass to read that _now_ also the first two seconds are important to you.