SPCC: Testrun of Rebel 16.1 finished

[pohl4711]

Ratinglist-testrun of Rebel 16.1 finished.


https://www.sp-cc.de

Also take a look at the EAS-Ratinglist, the world's first engine-ratinglist not measuring strength of engines but engines's style of play:
https://www.sp-cc.de/eas-ratinglist.htm

(Perhaps you have to clear your browsercache (press STRG+SHIFT+DEL) or reload the website))

[Modern Times]

No time forfeits for you ?

[Modern Times]

Ok, so you use a 2000ms (2 second) margin in cutechess which gives more headroom to engines that need it.

[pohl4711]

No time forfeits for you ?

Nope. timemargin is set to 2000ms in my testruns

[RubiChess]

That is a lot time margin.

And Rebel 16.1 is definitely buggy or at least interpretes the UCI wrong, means different to other engines and even more important different to cutechess. I hope that Chris and Ed are reading here...

The time forfeits in Rebel occure constantly when the increment is quite big (bigger than the timemargin). This would fit to Chris tests where he used very small increment and so a quite small timemargin was enough to avoid forfeit.

Here is an example of a forfeit that could easily be triggered. The game ran with concurrency 1 on a Ryzen3700x with nothing else running on it. So no delay by the OS or anything.
timemargin was 1000ms. Should be more than big enough according to Chris.
And the time control was 1+3 (increment 3 times bigger than timemargin!!!)

Complete command:

Code: Select all

cutechess-cli.exe -repeat -games 2 -rounds 50 -tournament gauntlet -pgnout rebel.pgn -concurrency 1 -openings file=UHO_XXL_+0.80_+1.09.epd format=epd order=random -engine name=RubiChess-20221229 cmd=RubiChess\cms20221229.exe proto=uci -engine name=Rebel-16.1 dir=Rebel cmd=rebel-16.1.exe proto=uci  -each option.Hash=64 option.Threads=1 tc=1.0+3 timemargin=1000 -debug  1>rebel-timetrouble-1+3.log

Log of the time forfeit:

Code: Select all

93942 >Rebel-16.1(1): go wtime 4000 btime 11114 winc 3000 binc 3000
93942 <Rebel-16.1(1): info depth 1 seldepth 2 time 1 nodes 4 pv c8b8 score cp -319 nps 4 tbhits 0 pvdepth 1
93942 <Rebel-16.1(1): info depth 2 seldepth 4 time 1 nodes 38 pv c8b8 d4d5 score cp -291 nps 38 tbhits 0 pvdepth 2
...
103481 <Rebel-16.1(1): info depth 22 seldepth 40 time 9540 nodes 19604602 pv d7b6 d1c1 e7e6 d4d5 c6d7 c4c5 e6d5 c5b6 c7b6 f1g2 e8d8 d3e5 d5e4 f3e4 c8c1 h1c1 d7e6 a3d3 b6b5 d3d8 f8d8 e5c6 d8d2 score cp -579 hashfull 917 nps 2054989 tbhits 0 pvdepth 23
106254 >Rebel-16.1(1): stop
106254 <Rebel-16.1(1): bestmove d7b6 
Finished game 1 (RubiChess-20221229 vs Rebel-16.1): 1-0 {Black loses on time}

106254-93942 = 12312ms (allowed: 11114ms), so 1198ms too late and bestmove was triggered by the GUI stop command!

So my guess is that Rebel adds the increment (binc 3000) to btime 11114 and thinks that it can use 14114ms for this move. But cutechess expects this move in 11114ms because it has already added the increment. You can see this by looking at the very first go that cutechess sends in a 1+3 game:

Code: Select all

2029 >RubiChess-20221229(0): go wtime 4000 btime 4000 winc 3000 binc 3000

So wtime value is already 1+3

Hope that helps improving.

Edit: My guess fits perfectly to what mwyoung writes in the ProDeo forum: time margin 500ms , increment 2000ms => time forfeits

Regards, Andreas

[Modern Times]

Edit: My guess fits perfectly to what mwyoung writes in the ProDeo forum: time margin 500ms , increment 2000ms => time forfeits
Regards, Andreas

Yes, I got lots of time forfeits with Rebel 16.1 under Cutechess GUI at the margin of 500ms that I've used for a couple of hundred thousand games with other engines without issue. This is the very first engine that has given me a problem. That is at 120+1 time control. Changing to 1000ms will probably solve that, and is what I'll end up doing, but I believe that the right solution is a change to the engine. Chris doesn't agree though.

[RubiChess]

Chris doesn't agree though.

Doesn't agree to what?
I checked Arena and we now have at least three common GUIs (Cutechess, Banksia, Arena) that expect the white engine to make the move in wtime and not in wtime+winc. And the UCI specs are quite clear imo:

* wtime <x>
white has x msec left on the clock

Regards, Andreas

[Modern Times]

He didn't agree that any changes were needed to Rebel (post on his forum). However now that you've identified more precisely what the issue really is, he may change his mind.

[chrisw]

That is a lot time margin.

And Rebel 16.1 is definitely buggy or at least interpretes the UCI wrong, means different to other engines and even more important different to cutechess. I hope that Chris and Ed are reading here...

The time forfeits in Rebel occure constantly when the increment is quite big (bigger than the timemargin). This would fit to Chris tests where he used very small increment and so a quite small timemargin was enough to avoid forfeit.

Here is an example of a forfeit that could easily be triggered. The game ran with concurrency 1 on a Ryzen3700x with nothing else running on it. So no delay by the OS or anything.
timemargin was 1000ms. Should be more than big enough according to Chris.
And the time control was 1+3 (increment 3 times bigger than timemargin!!!)

Complete command:

Code: Select all

cutechess-cli.exe -repeat -games 2 -rounds 50 -tournament gauntlet -pgnout rebel.pgn -concurrency 1 -openings file=UHO_XXL_+0.80_+1.09.epd format=epd order=random -engine name=RubiChess-20221229 cmd=RubiChess\cms20221229.exe proto=uci -engine name=Rebel-16.1 dir=Rebel cmd=rebel-16.1.exe proto=uci  -each option.Hash=64 option.Threads=1 tc=1.0+3 timemargin=1000 -debug  1>rebel-timetrouble-1+3.log

Log of the time forfeit:

Code: Select all

93942 >Rebel-16.1(1): go wtime 4000 btime 11114 winc 3000 binc 3000
93942 <Rebel-16.1(1): info depth 1 seldepth 2 time 1 nodes 4 pv c8b8 score cp -319 nps 4 tbhits 0 pvdepth 1
93942 <Rebel-16.1(1): info depth 2 seldepth 4 time 1 nodes 38 pv c8b8 d4d5 score cp -291 nps 38 tbhits 0 pvdepth 2
...
103481 <Rebel-16.1(1): info depth 22 seldepth 40 time 9540 nodes 19604602 pv d7b6 d1c1 e7e6 d4d5 c6d7 c4c5 e6d5 c5b6 c7b6 f1g2 e8d8 d3e5 d5e4 f3e4 c8c1 h1c1 d7e6 a3d3 b6b5 d3d8 f8d8 e5c6 d8d2 score cp -579 hashfull 917 nps 2054989 tbhits 0 pvdepth 23
106254 >Rebel-16.1(1): stop
106254 <Rebel-16.1(1): bestmove d7b6 
Finished game 1 (RubiChess-20221229 vs Rebel-16.1): 1-0 {Black loses on time}

106254-93942 = 12312ms (allowed: 11114ms), so 1198ms too late and bestmove was triggered by the GUI stop command!

So my guess is that Rebel adds the increment (binc 3000) to btime 11114 and thinks that it can use 14114ms for this move. But cutechess expects this move in 11114ms because it has already added the increment. You can see this by looking at the very first go that cutechess sends in a 1+3 game:

Code: Select all

2029 >RubiChess-20221229(0): go wtime 4000 btime 4000 winc 3000 binc 3000

So wtime value is already 1+3

Hope that helps improving.

Edit: My guess fits perfectly to what mwyoung writes in the ProDeo forum: time margin 500ms , increment 2000ms => time forfeits

Regards, Andreas

Oh what a pig, they already added the increment?
Re-write .......

[Rebel]

He didn't agree that any changes were needed to Rebel (post on his forum). However now that you've identified more precisely what the issue really is, he may change his mind.

Yes, we will.

Thanks Andreas for pointing out [!]