cutechess-cli connection stalls

[jswaff]

I'm trying to do some round robin testing using (the newest) cutechess-cli.

Quite often after several hundred games the entire thing stops with an error like 'White's connection stalls'. I think it may be specific to certain engines. It seems to happen with Beowulf and my own engine quite a bit.

I'm testing at 3 + 0.3. Maybe a longer TC would help but that would slow the whole process down. Is there anything else I can try?

The testing machine is an older laptop with a fresh install of Windows 7, if that matters.

--
James

[Sven]

I'm trying to do some round robin testing using (the newest) cutechess-cli.

Quite often after several hundred games the entire thing stops with an error like 'White's connection stalls'. I think it may be specific to certain engines.

I saw exactly this message (for White or Black) with cutechess-cli 0.8.2, but only if I run concurrent games (with the option "-concurrency N"), and then it seems to be independent from given engines. So are you running concurrent games, or only one at a time?

[jswaff]

Just one at a time. It's an older laptop with only one physical core (two logical cores). I just started another run with '-wait 1 ' to see if that makes any difference.

[ZirconiumX]

Just one at a time. It's an older laptop with only one physical core (two logical cores). I just started another run with '-wait 1 ' to see if that makes any difference.

There's your problem then. You're running everything on one core, so Windows has to interrupt your engine to run its background tasks (like the GUI). However, because cutechess measures by wall-clock time, this results in a time loss/connection stall because the program isn't running when the time runs out.

[jswaff]

If it uses wall clock then maybe that TC is just too fast for that poor laptop. Though, it is a fresh installation of Windows with nothing else installed- there shouldn't be too much going on in the background. I wonder how long an engine has to be non responsive to trigger a "connection stalled" error.

[hgm]

The problem was not time forfeits, but connection stalls. Which means a hanging or crashed engine.

[jswaff]

Right. I guess I need to figure out if its really specific to certain engines. If so then the solution is obvious. If not...

[Ferdy]

I'm trying to do some round robin testing using (the newest) cutechess-cli.

Quite often after several hundred games the entire thing stops with an error like 'White's connection stalls'. I think it may be specific to certain engines. It seems to happen with Beowulf and my own engine quite a bit.

I'm testing at 3 + 0.3. Maybe a longer TC would help but that would slow the whole process down. Is there anything else I can try?

The testing machine is an older laptop with a fresh install of Windows 7, if that matters.

--
James

Is that 3s + 0.3s? if so then it is fast indeed.

Probably most old winboard engines does not support increments below 1 sec. I guess Beowulf is a winboard engine.

In case you are not aware, you can use the option

Code: Select all

-recover

so that cutechess-cli will restart the stalled engine and hope that the test would continue. Then inspect later the resulting pgn.

To see the interaction between engines and cli, useful for debugging, you may use the option with redirection, works for me in windows.

Code: Select all

-debug >debugfile.txt

If you are into debugging mode, better to use winboard.

The latest cutechess-cli that I am using now is,

Code: Select all

cutechess-cli 0.9.4.1877

I think the latest release version was cutechess-cli 0.8.5.

[jswaff]

Thanks Ferdinand, that's very helpful. It's 2200 games into a 2400 game run right now. Using '-wait 1' seems to have done the trick, but I'll do some more testing and will keep everything you suggested in mind. Greatly appreciated.

[jswaff]

Interestingly the '-wait 1' option resolved the "connection stalled" error. It made it through an entire 2400 game round robin. However, Mediocre and Beowulf both have a lot of time losses so I guess the bottom line is that the TC is just too fast for those engines on that hardware.

I'm not real surprised by mediocre, being a JVM based engine.