Additional UCI output registering as massive elo losses

[AndrewGrant]

Currently Ethereal only prints information upon completing an iteration of the main iterative deepening loop. This includes the depth, the score, the nodes, the nps, the hashfull, and the PV.

Recently I've been trying to use Ethereal to do some analysis (following TCEC games and what now) and would like to include seldepth, printing on upper/lower bounds, and printing mate scores instead of just printing a massive number.

For some reason, every single on of these attempts is a massive elo loser in a quick SPRT test.

Here is one such attempt at adding more pv functionality
https://github.com/AndyGrant/EtherealDe ... 6136fa0fcd


Obviously more printing means more time usage, so I should expect some elo loss. But I'm finding losses of 10,15+ elo.

I'll not that in order to make Ethereal work with GUI's like Arena, I do an fflush(stdout) with each new line of UCI output.
Removing this line seems to (break Arena, clearly) and be elo neutral ( W: 960 L: 964 D: 1726 ) @ 3s+.03s

Finally, I do testing through cutechess, and I do not have cutechess saving debug or PGN files.

Any thoughts? I've run out of ideas here.

[elcabesa]

at very fast time control every UCI output is a big ELO loss from my experience.

this effect is mitigated with longer time control. What time control do you use for SPRT test?

[hgm]

10 Elo corresponds to ~10% slowdown. At ultra-fast TC that sounds believable; system functions are usually expensive. At long TC the slowdown would probably be negligible, and there would be no effect on Elo at all.

One option is to simply ignore the Elo loss, as in a sense it is an artifact of the fast TC, an no one is really interested in using the engine at this TC.

Many UCI engines cut down on output durig the first 10 sec of the search. You probably don't want to do that for PVs, but you could still cut down on flushing, and suppress that during the first 10 sec (but always do it after 'bestmove').

[cdani]

...
Many UCI engines cut down on output durig the first 10 sec of the search. You probably don't want to do that for PVs, but you could still cut down on flushing, and suppress that during the first 10 sec (but always do it after 'bestmove').

+1 Nobody will use such initial information, that is overwritten so fast.

[hgm]

Well, as an engine author it is sometimes interesting for me what the engine was thinking at the lowest depth, and how many ply exactly it took before it saw some tactics. Even if this would just be 4 or 5 ply.

But it would not matter to me whether I see that within 1 sec. So flushing the PVs only after some minimal time has elapsed would be acceptable. Perhaps it woul be good to flush at least one output line immeiately: it is a bit disconcerting when you expect the engine to start searching, and you don't see anything printed at all for a number of seconds. (I know, because this is what my engine used to do after a ponder hit!)

[Ozymandias]

One option is to simply ignore the Elo loss, as in a sense it is an artefact of the fast TC, an no one is really interested in using the engine at this TC.

Call me "no one", 231 million games and counting.

[Sven]

in order to make Ethereal work with GUI's like Arena, I do an fflush(stdout) with each new line of UCI output.
Removing this line seems to (break Arena, clearly) and be elo neutral

I think flushing output is only important for those lines that are relevant for the protocol. E.g. if you do not flush the "bestmove" output then you might lose on time. But flushing PVs and other info lines that are printed during search should not be necessary, maybe with the exception of the final PV (if the engine explicitly prints one at the very end of the search).

[AndrewGrant]

I believe you would be correct.

However, it is nice to flush so that GUI's like Arena can show what the engine is thinking.

Anyway, this whole thing is a bit of a non issue. Just wanted to make sure that this struggle was not just my own

[AndrewGrant]

One option is to simply ignore the Elo loss, as in a sense it is an artifact of the fast TC, an no one is really interested in using the engine at this TC.

I think that is what I will be doing. No (super good) reason to try to be the best engine at 5+.05s

I have tried the cutting down on UCI output as you suggested -- surprisingly even with games at 5+.05s I found the difference to be about the same as no UCI flushing, and about the same as standard outputting.

Anyway, I have been reassured with the quote "System functions are usually expensive", which is what I expected / hoped.

Thanks for your insights

[hgm]

I think flushing output is only important for those lines that are relevant for the protocol. E.g. if you do not flush the "bestmove" output then you might lose on time. But flushing PVs and other info lines that are printed during search should not be necessary, maybe with the exception of the final PV (if the engine explicitly prints one at the very end of the search).

It is quite annoying when the engine thinks (perhaps for a few minutes) an you don't get to see anything at all, an then, when it finally moves, the PVs of all iterations appear at once. For interactive analysis this would even be intolerable.