c-chess-cli

[lucasart]

Use pipe2() instead of pipe(), with flag=O_CLOEXEC, to atomically set the FD_CLOEXEC flags when creating the pipe. This only works on Linux

Works so far, I'm running commit 285 under Linux. The engine processes also have the EPD and PGN files open, which they wouldn't with the pure POSIX solution, but I don't think that this is an issue. Also, the performance impact is probably smaller than it may look because you already avoid spawning engine processes if no engine switch is required.

I implemented sequential writing for PGN file (and sample file if you care). PGN games are now always written in the correct order, rather than in random order as threads produce them.

Of course, this comes at the cost of shared buffer, guarded by a mutex. Not measurable in my experience, but I only have 8 threads (4 core i7).

I wonder how it fares in your 285 threads situation.

[Ras]

Of course, this comes at the cost of shared buffer, guarded by a mutex. Not measurable in my experience, but I only have 8 threads (4 core i7).

I think it would require pretty short time controls and many threads to be measurable.

I wonder how it fares in your 285 threads situation.

I run only 8 games in parallel on my 4C/8T CPU. The 285 was the Github commit number of c-chess-cli (sort of a version number, now at 300), not the thread count.

[Ras]

New feature idea: pausing tournaments. That would be useful for doing something else that needs processor power in-between, but without aborting the tournament. It could work as follows:

• Upon program start, c-chess-cli could print "press 'p' to pause tournament"
• Pressing the 'p' key for "pause"
• c-chess-cli prints "finishing ongoing games..."
• the ongoing games are finished, but no new ones started
• once all ongoing games have been finished, c-chess-cli prints "tournament paused, press 'r' to resume."

• Restarting would work via pressing 'r'.
• c-chess-cli would print "tournament resumed." and start the next games.

A different key for pause and resume would prevent the user from accidentally double-hitting the pause key and then being unclear about the state. Pressing 'r' before all ongoing games have been finished would just remove the pause state along with printing the "tournament resumed" message.

[hgm]

A generalization of this would be interactively modifying the concurrency. Setting it to 0 would pause the tournament. But you could then make exactly enough system resources available to do what you wanted to do on the side, by typing any other number. That brings it on par with XBoard, where you can also close as many of the workers as you want, to lower concurrency or pause the tourney completely.

[Ras]

A generalization of this would be interactively modifying the concurrency.

Also interesting, though pause/resume would be easier to implement - leaving a possible generalisation for later.

[hgm]

I am not sure whether it would be much different. I suppose the pausing would still finish ongoing games. (You could of course also have a 'hard' pausing mode, which aborts all ongoing games; in XBoard you select that on a per-game basis, by either just quitting an XBoard worker, or switching off its tournament mode.)

So in both cases the implementation would be to test after a game finishes whether a new game should be started. With all-or-nothing pausing you would do this by testing a pause flag for being 0 or 1. For adjusting concurrency you would test whether the current number of games is below the maximum or not. Hardly a difference. Except that you would have to keep track of the number of currently running games. But I suppose that is already done, and even if not, it seems quite trivial.

[lucasart]

New feature idea: pausing tournaments. That would be useful for doing something else that needs processor power in-between, but without aborting the tournament. It could work as follows:

• Upon program start, c-chess-cli could print "press 'p' to pause tournament"
• Pressing the 'p' key for "pause"
• c-chess-cli prints "finishing ongoing games..."
• the ongoing games are finished, but no new ones started
• once all ongoing games have been finished, c-chess-cli prints "tournament paused, press 'r' to resume."

• Restarting would work via pressing 'r'.
• c-chess-cli would print "tournament resumed." and start the next games.

A different key for pause and resume would prevent the user from accidentally double-hitting the pause key and then being unclear about the state. Pressing 'r' before all ongoing games have been finished would just remove the pause state along with printing the "tournament resumed" message.

Another use case is system crashes, or computer switches off (eg. overheating). Or even a power cut. Could be the grid. Or your wife accidently stumbles on the wire while vacuming

So I would rather make the state persist in a tournament file, and be able to resume from that. This will also be much easier to implement than the interactive logic you propose (pausing/resuming threads is easier said than done, don't forget all these threads are busy waiting for blocking I/O). All I need to do is write a file with the argument list, the idx of the next game to play, and the (W,L,D) triplets for each pair.

The difficulty is that ongoing and mis-ordered games are lost, and can't be counted in the idx value. So I must only count the longest sequential sequence of completed games. For example, if we played (0,1,2,5,7,6,3), then (0,1,2,3) can be counted, and the tournament should resume with idx=4. Fortunately, I already have that logic built for sequential PGN output, so I can try to reuse it.

Regarding HGM's flexible concurrency, I think it could be viewed as a natural extension of this. You could specify the concurrency as you resume, overriding the original value. So you start with 8, then hit Ctrl+C, then resume with 6, etc. I would rather do it that way, than changing concurrency on the fly with the program still running, which is over-engineneering IMO.

[hgm]

So I would rather make the state persist in a tournament file, and be able to resume from that.

That is how I do it in XBoard. A tournament is a file, describing all tourney parameters, and the results so far are at the end, so new results can easily be added as they are produced. (Game records go to a different file.) That way you can resume a tourney after a power-down, and even play other tourneys in between, (or in parallel), from a different tourney file.

Difference is that there never is any concurrency within XBoard; the concurrency is achieved by putting multiple XBoard instances at work on the same tourney file. There are safeguards to protect the critical code section where these XBoard 'workers' have to start a new game, to prevent two of them grabbing the same game.

[Ras]

Feature suggestion: the "illegal move in PV" warning should include which side (or which engine) is sending the bad PV. I'd like to quickly see whether it's my engine for deciding whether I should investigate. Right now, that is only available if full logging is active, but I don't want to log some 100k games just in case.

[lucasart]

Feature suggestion: the "illegal move in PV" warning should include which side (or which engine) is sending the bad PV. I'd like to quickly see whether it's my engine for deciding whether I should investigate. Right now, that is only available if full logging is active, but I don't want to log some 100k games just in case.

Agreed. Fixed now.