Currently Ethereal only prints information upon completing an iteration of the main iterative deepening loop. This includes the depth, the score, the nodes, the nps, the hashfull, and the PV.
Recently I've been trying to use Ethereal to do some analysis (following TCEC games and what now) and would like to include seldepth, printing on upper/lower bounds, and printing mate scores instead of just printing a massive number.
For some reason, every single on of these attempts is a massive elo loser in a quick SPRT test.
Here is one such attempt at adding more pv functionality
https://github.com/AndyGrant/EtherealDe ... 6136fa0fcd
Obviously more printing means more time usage, so I should expect some elo loss. But I'm finding losses of 10,15+ elo.
I'll not that in order to make Ethereal work with GUI's like Arena, I do an fflush(stdout) with each new line of UCI output.
Removing this line seems to (break Arena, clearly) and be elo neutral ( W: 960 L: 964 D: 1726 ) @ 3s+.03s
Finally, I do testing through cutechess, and I do not have cutechess saving debug or PGN files.
Any thoughts? I've run out of ideas here.
Additional UCI output registering as massive elo losses
Moderators: hgm, Rebel, chrisw
-
- Posts: 1756
- Joined: Tue Apr 19, 2016 6:08 am
- Location: U.S.A
- Full name: Andrew Grant
Additional UCI output registering as massive elo losses
#WeAreAllDraude #JusticeForDraude #RememberDraude #LeptirBigUltra
"Those who can't do, clone instead" - Eduard ( A real life friend, not this forum's Eduard )
"Those who can't do, clone instead" - Eduard ( A real life friend, not this forum's Eduard )
-
- Posts: 855
- Joined: Sun May 23, 2010 1:32 pm
Re: Additional UCI output registering as massive elo losses
at very fast time control every UCI output is a big ELO loss from my experience.
this effect is mitigated with longer time control. What time control do you use for SPRT test?
this effect is mitigated with longer time control. What time control do you use for SPRT test?
-
- Posts: 27808
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: Additional UCI output registering as massive elo losses
10 Elo corresponds to ~10% slowdown. At ultra-fast TC that sounds believable; system functions are usually expensive. At long TC the slowdown would probably be negligible, and there would be no effect on Elo at all.
One option is to simply ignore the Elo loss, as in a sense it is an artifact of the fast TC, an no one is really interested in using the engine at this TC.
Many UCI engines cut down on output durig the first 10 sec of the search. You probably don't want to do that for PVs, but you could still cut down on flushing, and suppress that during the first 10 sec (but always do it after 'bestmove').
One option is to simply ignore the Elo loss, as in a sense it is an artifact of the fast TC, an no one is really interested in using the engine at this TC.
Many UCI engines cut down on output durig the first 10 sec of the search. You probably don't want to do that for PVs, but you could still cut down on flushing, and suppress that during the first 10 sec (but always do it after 'bestmove').
-
- Posts: 2204
- Joined: Sat Jan 18, 2014 10:24 am
- Location: Andorra
Re: Additional UCI output registering as massive elo losses
+1 Nobody will use such initial information, that is overwritten so fast.hgm wrote:...
Many UCI engines cut down on output durig the first 10 sec of the search. You probably don't want to do that for PVs, but you could still cut down on flushing, and suppress that during the first 10 sec (but always do it after 'bestmove').
Daniel José - http://www.andscacs.com
-
- Posts: 27808
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: Additional UCI output registering as massive elo losses
Well, as an engine author it is sometimes interesting for me what the engine was thinking at the lowest depth, and how many ply exactly it took before it saw some tactics. Even if this would just be 4 or 5 ply.
But it would not matter to me whether I see that within 1 sec. So flushing the PVs only after some minimal time has elapsed would be acceptable. Perhaps it woul be good to flush at least one output line immeiately: it is a bit disconcerting when you expect the engine to start searching, and you don't see anything printed at all for a number of seconds. (I know, because this is what my engine used to do after a ponder hit!)
But it would not matter to me whether I see that within 1 sec. So flushing the PVs only after some minimal time has elapsed would be acceptable. Perhaps it woul be good to flush at least one output line immeiately: it is a bit disconcerting when you expect the engine to start searching, and you don't see anything printed at all for a number of seconds. (I know, because this is what my engine used to do after a ponder hit!)
-
- Posts: 1535
- Joined: Sun Oct 25, 2009 2:30 am
Re: Additional UCI output registering as massive elo losses
Call me "no one", 231 million games and counting.him wrote:One option is to simply ignore the Elo loss, as in a sense it is an artefact of the fast TC, an no one is really interested in using the engine at this TC.
-
- Posts: 4052
- Joined: Thu May 15, 2008 9:57 pm
- Location: Berlin, Germany
- Full name: Sven Schüle
Re: Additional UCI output registering as massive elo losses
I think flushing output is only important for those lines that are relevant for the protocol. E.g. if you do not flush the "bestmove" output then you might lose on time. But flushing PVs and other info lines that are printed during search should not be necessary, maybe with the exception of the final PV (if the engine explicitly prints one at the very end of the search).AndrewGrant wrote:in order to make Ethereal work with GUI's like Arena, I do an fflush(stdout) with each new line of UCI output.
Removing this line seems to (break Arena, clearly) and be elo neutral
-
- Posts: 1756
- Joined: Tue Apr 19, 2016 6:08 am
- Location: U.S.A
- Full name: Andrew Grant
Re: Additional UCI output registering as massive elo losses
I believe you would be correct.
However, it is nice to flush so that GUI's like Arena can show what the engine is thinking.
Anyway, this whole thing is a bit of a non issue. Just wanted to make sure that this struggle was not just my own
However, it is nice to flush so that GUI's like Arena can show what the engine is thinking.
Anyway, this whole thing is a bit of a non issue. Just wanted to make sure that this struggle was not just my own
Last edited by AndrewGrant on Wed Apr 11, 2018 10:14 pm, edited 1 time in total.
#WeAreAllDraude #JusticeForDraude #RememberDraude #LeptirBigUltra
"Those who can't do, clone instead" - Eduard ( A real life friend, not this forum's Eduard )
"Those who can't do, clone instead" - Eduard ( A real life friend, not this forum's Eduard )
-
- Posts: 1756
- Joined: Tue Apr 19, 2016 6:08 am
- Location: U.S.A
- Full name: Andrew Grant
Re: Additional UCI output registering as massive elo losses
I think that is what I will be doing. No (super good) reason to try to be the best engine at 5+.05sOne option is to simply ignore the Elo loss, as in a sense it is an artifact of the fast TC, an no one is really interested in using the engine at this TC.
I have tried the cutting down on UCI output as you suggested -- surprisingly even with games at 5+.05s I found the difference to be about the same as no UCI flushing, and about the same as standard outputting.
Anyway, I have been reassured with the quote "System functions are usually expensive", which is what I expected / hoped.
Thanks for your insights
#WeAreAllDraude #JusticeForDraude #RememberDraude #LeptirBigUltra
"Those who can't do, clone instead" - Eduard ( A real life friend, not this forum's Eduard )
"Those who can't do, clone instead" - Eduard ( A real life friend, not this forum's Eduard )
-
- Posts: 27808
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: Additional UCI output registering as massive elo losses
It is quite annoying when the engine thinks (perhaps for a few minutes) an you don't get to see anything at all, an then, when it finally moves, the PVs of all iterations appear at once. For interactive analysis this would even be intolerable.Sven wrote:I think flushing output is only important for those lines that are relevant for the protocol. E.g. if you do not flush the "bestmove" output then you might lose on time. But flushing PVs and other info lines that are printed during search should not be necessary, maybe with the exception of the final PV (if the engine explicitly prints one at the very end of the search).