UCI Win/Draw/Loss reporting

[Gian-Carlo Pascutto]

I suggest a simple way: engines just send (JSON info) what they want.

This won't work for reasons already outlined in my post. The engine has no idea how much detail the GUI wants. And you can send enough info the make the GUI crawl...

HGM's idea seems fine.

[Gian-Carlo Pascutto]

This tells the engine what extra infos it can put in an 'info' command that the GUI knows how to display. If the GUI says it understands 'wdl', the engine can put wdl info with its PV, otherwise it will refrain from doing so. (Or it will find some backward-compatible way to display the info, like 'info string'). It is up to the community to negociate what infos they want GUIs to display, and what keyword to define for the ability to do so.

How do you feel about using JSON (as crem suggested) over UCI-style tag info?

I'm not jumping with enthusiasm to add JSON to UCI, but it does seem like it would solve the extensibility issue, whereas including extra info tags could make robust parsing very hard very quickly.

Edit: The latter is especially true if we envisage the info containing a tree!

[phhnguyen]

Let me put it differently: with some more work (i.e. thinking about enable/disable info issue) it might be possible to specify a way to send more info inside UCI that could contain WDL and also serve as a well-designed vehicle for transmitting more information (which still has to be specified, because JSON by itself doesn't solve that).

I suggest instead of focusing on WDL only, we may design a kind of self-explain data, in which engines can tell chess GUIs how to display their data.

For example, to display a table (of WDL):

Code: Select all

uci json {"table": ["W": 60, "D": 20, "L": 20]}

To display arrows:

Code: Select all

uci json {"arrows": [{cells: "a1c3", "color": 0xff00},  {cells: "d7c3", "color": 0x00ff00}}

To display bubbles (circles with info in celles):

Code: Select all

uci json {"bubbles": [{cell: "a1", "color": 0xff00, "values": [3, 5]}}

It may be complicated for the first time but programmers can learn, try-error and practice from small, easy to large, hard cases.

But introducing JSON seems like a large barrier for just getting WDL info. So I'm not sure this is a good carrot to get GUI people to implement something.

For me, a chess-hobby programmer, "a good carrot" is that my freeware will be useful for many people. More challenge is actually more interest. If you think that is useful for the long term, go ahead, we will sooner or later support you.

[phhnguyen]

I suggest a simple way: engines just send (JSON info) what they want.

This won't work for reasons already outlined in my post. The engine has no idea how much detail the GUI wants. And you can send enough info the make the GUI crawl...

HGM's idea seems fine.

That is why I agreed with your suggestion:

Code: Select all

option name UCI_JSONExtension type check default false

If an engine sends too much data to slowdown and/or crash chess GUIs, clearly it has a serious bug

[Ferdy]

Someone pointed me a while ago (I forgot where, might've been TCEC chat) to this: https://github.com/DanielUranga/lc0/com ... 0df2f06db2
Which is allegedly how Fat Fritz does WDL reporting?

I have no idea if this is accurate, but in general I feel it would be useful to standardize some format if we want to report this information to interfaces. Based on the above patch, I suggest:

option name UCI_ShowWDL type check default false

If this is toggled to enabled, the engine will include wdl info in the info updates. Making it optional prevents interfaces from breaking, even though UCI does say the interface should ignore stuff it doesn't understand. But depending on the option seems safer as UCI is widely deployed and currmove etc use the same technique. GUIs which don't know about the option (and can't show wdl anyway) won't show it, if they're correctly implemented (UCI spec says UCI_ options shouldn't be displayed to the user).

For the wdl reporting itself I will use the format from the above patch, i.e.:

info depth 16 seldepth 62 time 11780 nodes 65546 tbhits 0 score cp 25 wdl 210 608 181 nps 5564 hashfull 201 multipv 1 pv d2d4 g8f6 c2c4 e7e6 g1f3 d7d5 b1c3 f8b4 c1g5 h7h6 g5f6 d8f6 d1a4 b8c6 e2e3 e8g8 f1e2 c8d7 a4b3 d5c4 b3c4 f6e7 e1g1 b4d6 c3e4 e6e5 d4d5 c6d8

So

wdl win_probability_in_permille draw_probability_in_permille loss_probability_in_permille

In the above example, win probability = 21.0%, draw probability = 60.8% and loss probability = 18.1%. Note that due to rounding they might not sum to 100% exactly.

Engines that don't have an explicitly draw probability calculation can still obtain a good estimate from looking at the absolute score (i.e. say 65% at eval = 0 and 1% at eval = +300 with interpolation, to give a random example) so using this format shouldn't block them. (And really, draw probabilities are very useful to users I think...)

Unless there's a very compelling reason to not use this format, I'll go ahead and use this in Stoofvlees, and would encourage others to do the same.

Perhaps simpler and more uci way of communicating is by key value pair. Example,

Code: Select all

winprob 20 drawprob 60 lossprob 20

values like 20, 60 and 20 are in percent.

On winning position one may just send winprob 99 and ignore draw and loss prob.

If position is drawish,
drawprob 100 and skip sending winprob and lossprob.

Another option is to put those under score info. Example,

Code: Select all

score winprob 20 score drawprob 60 score lossprob 20

Another method of sending those info is just to use info string format. Example,

Code: Select all

info string winprob 20 drawprob 60 lossprob 20

wdl does not always change, when there are significant changes, the engine may send it.
Chessbase Fritz GUI has a specific info box designed for info string, separate from the more usual info depth ... score cp ... pv stuffs. I think it is engine chatter box. This box has potential to be very useful when an engine someday will be able to send descriptive info to the user. Example,

Code: Select all

info string I don't like your bishop location currently pointing to my king, will terminate it in 3 moves.

When user looks at the pv, there is a move that captures a bishop.

[Gian-Carlo Pascutto]

Perhaps simpler and more uci way of communicating is by key value pair. Example,

Code: Select all

winprob 20 drawprob 60 lossprob 20

values like 20, 60 and 20 are in percent.

I don't see the benefit of having extra tags. We don't send a pv either with pv e4e4 pv e7e5 pv g1f3. I also don't see how it is simpler, looks the opposite to me.

On winning position one may just send winprob 99 and ignore draw and loss prob.
If position is drawish,
drawprob 100 and skip sending winprob and lossprob.

This puts the burden of inventing something meaningful there to the GUI, which seems bad.

Another option is to put those under score info. Example,

Code: Select all

score winprob 20 score drawprob 60 score lossprob 20

I wondered whether it should be written as score wdl 20 50 39, because the UCI protocol has some options there, but it's not clear why the grouping around score exists. I don't think wdl is exclusive to providing cp OR mate, so I explicitly avoided saying the grouping is required.

Another method of sending those info is just to use info string format.

This is freeformat and engines already put freeformat stuff there so that would be a compatibility nightmare if the GUI has to go and hunt for something meaningful in there. If you agree that it only does that when some setoption is done we're basically where we are now except there's a proposal to use JSON. There's no need for backwards compat if the setoption exists, and if the setoption doesn't exist then existing GUIs will be overflowed with garbage. So not a good idea.

[Gian-Carlo Pascutto]

If an engine sends too much data to slowdown and/or crash chess GUIs, clearly it has a serious bug

Who has what bug?

It's reasonable to optionally want the engine to dump out a ton of info every move for debugging. If there's no way to communicate what is useful to the UCI partner, the situation described is unavoidable and it has nothing to do with "bugs".

[hgm]

I very much dislike the JSON. The PV in an info command is also not sent as a JSON array. I don't see any advantage to it other than uglifying the output. If you want to send a move tree, you can use parentheses like in PGN recursive variations.

[Ras]

Why that configuration stuff at all? Just include the WDL with the three values for win, draw, loss in the info from engine to GUI. UCI explicitely specifies ignoring unknown things and continue parsing. If a GUI barfs on the wdl inside the info, it is not UCI-compliant anyway and should be fixed:

* if the engine or the GUI receives an unknown command or token it should just ignore it and try to parse the rest of the string in this line.
Examples: "joho debug on\n" should switch the debug mode on given that joho is not defined, "debug joho on\n" will be undefined however.

The only question is whether to use permill or percent. Given that hashfull and cpuload both report in permill, wdl should also use permill for consistency.

One UCI tag having three info parameters isn't uncommon - examples are pv and searchmoves.

[hgm]

That is true if it was just about wdl. But, as already remarked, if we would get dozens of new infos, or very verbose infos, it would be nice if the GUI would be able to selectively switch them on or off.