new project: CLI for chess tournaments

[lucasart]

but I find fork() to be much easier (if a bit dirty) to be honest...

fork is not dirty at all. It is a very clean way to start up processes.

I have to say, I found fork() quite weird. And it took me some time to figure out how it actually works. But I don't know of another way to do it.

[lucasart]

I suspect it won't compile on Windows though, since I used fork() to split off the engine processes. I could switch to threads (I've used pthreads in the past), but I find fork() to be much easier (if a bit dirty) to be honest...

Not sure what you mean. The fork() and exec() system calls are used to spawn child processes (engines). Threads are just threads of your process. So two completely different things, no ?

[lucasart]

I'm trying to do something like cutechess-cli. For the moment, it's very limited, but some of the basics are there (can play a single game, UCI only, POSIX only).

Out of curiosity, is it just your own hobby project, or do you hope to fill a need that cutechess-cli doesn't fill?

It's really just a hobby project. Nothing big and professional like cutechess. If I can do the basic functionalities of cutechess, cutting corners along the way, I'll be happy already.

[lucasart]

The makefile did not work for me.

I replaced the compile command by

g++ *.cc -o ./chess -DNDEBUG -Wall -std=gnu++0x -g -O2

which worked.

The executable seems to run fine.

std=c++11 is probably only available on recent versions of GCC. I'm using GCC 4.7.2 at the moment.

[Michel]

The fork() and exec() system calls are used to spawn child processes (engines). Threads are just threads of your process. So two completely different things, no ?

Fork and threads are both wrappers around the clone system call I believe.

clone() creates a new process, in a manner similar to fork(2). It is
actually a library function layered on top of the underlying clone()
system call, hereinafter referred to as sys_clone. A description of
sys_clone is given towards the end of this page.

Unlike fork(2), these calls allow the child process to share parts of
its execution context with the calling process, such as the memory
space, the table of file descriptors, and the table of signal handlers.
(Note that on this manual page, "calling process" normally corresponds
to "parent process". But see the description of CLONE_PARENT below.)

The main use of clone() is to implement threads: multiple threads of
control in a program that run concurrently in a shared memory space.

[lucasart]

The fork() and exec() system calls are used to spawn child processes (engines). Threads are just threads of your process. So two completely different things, no ?

Fork and threads are both wrappers around the clone system call I believe.

clone() creates a new process, in a manner similar to fork(2). It is
actually a library function layered on top of the underlying clone()
system call, hereinafter referred to as sys_clone. A description of
sys_clone is given towards the end of this page.

Unlike fork(2), these calls allow the child process to share parts of
its execution context with the calling process, such as the memory
space, the table of file descriptors, and the table of signal handlers.
(Note that on this manual page, "calling process" normally corresponds
to "parent process". But see the description of CLONE_PARENT below.)

The main use of clone() is to implement threads: multiple threads of
control in a program that run concurrently in a shared memory space.

So as I understand clone() is like pthread_create() ?

Let's say you create a new thread. And then what do you do with that ? You will need to call an exec() system call, which will overwrite the code of your process no ?

Besides, I don't really understand the point of threads here. Why would you share your memory pages with the child processes (or threads assuming you can do it with threads) ?

Edit: Had a look at the clone() documentation. I understand a bit more now. But I'm not convinced it's a good idea to share my memory tables, file descriptors, and signal handlers with the child process. At lease fork() will ring-fence my memory from buggy engines (child processes). The file descriptors, I'm not sure what will happen, but I need everyone to have their own stdin/stdout and all of these are different per process. Anyway, I'd be glad if someone could show me with actual code what you mean by using threads instead of fork().

[lucasart]

I'm trying to do something like cutechess-cli. For the moment, it's very limited, but some of the basics are there (can play a single game, UCI only, POSIX only).

I'm already following your project at Github and I wish you success.

Thank you!

I remember when cutechess-cli was at that stage - you have quite a lot of work ahead of you.

I know... But unlike you, I will be cutting corners along the way, big time ! For example:
* I will never do a PGN parser, or a Polyglot book reader. Instead I will just do an EPD reader (store the FENs in a vector<string> and select randomly or sequentially, nothing fancier than that).
* I will never do any chess variant. Shuffle chess will be implicitly supported by setting FEN positions into the EPD book. I'll probably do Chess960 at a later stage, if and when all the basics are done.
* I'll probably just do 2 engine match (number of games, EPD file sequential or random, time controls, depth, nodes etc). And to play tournaments (gauntlets or round-robin), I will probably make a Python script that calls my C++ program for each pair. Same thing for playing games in parallel. None of that game scheduling is really performance critical anyway. That being said, the actual process communication, UCI parsing and "game referee" things, are done with performance in mind.

[Michel]

Sorry I was just replying to your comment that forked processes and threads are totally different things. On Linux they are in fact the same (but hidden under different wrappers).

[Evert]

I know... But unlike you, I will be cutting corners along the way, big time ! For example:
* I will never do a PGN parser, or a Polyglot book reader. Instead I will just do an EPD reader (store the FENs in a vector<string> and select randomly or sequentially, nothing fancier than that).

Same here.
I might look into polyglot books though. Might, depending on how easy it is to get my hands on something for variants and have that work well with Sjaak.

* I will never do any chess variant. Shuffle chess will be implicitly supported by setting FEN positions into the EPD book. I'll probably do Chess960 at a later stage, if and when all the basics are done.

Quite the opposite in my case.
I'm not really interested in Chess960, but I should probably hack that in at some point anyway. I'm more interested in Seirawan, but that'll be a bit more tricky because Sjaak at the moment can't handle it.

* I'll probably just do 2 engine match (number of games, EPD file sequential or random, time controls, depth, nodes etc). And to play tournaments (gauntlets or round-robin), I will probably make a Python script that calls my C++ program for each pair. Same thing for playing games in parallel. None of that game scheduling is really performance critical anyway. That being said, the actual process communication, UCI parsing and "game referee" things, are done with performance in mind.

Same again here. Sjaak is actually not very fast but Sjef stops the engine clocks before making doing legality checking and claim verification so it doesn't "steal" time from the engines. So far it's been working really well, and I've found a couple of bugs in both Sjaak and Leonidas while working on Sjef.

[lucasart]

OK, I can now play a head to head match. Example:

Code: Select all

lucas@lucas-desktop:~/Chess/chess$ ./chess ../Engines/discocheck_3.7.1 ../Engines/stockfish_2.3.1 ../test.epd 10
0 - 0 - 1	Draw by 3-fold repetition
0 - 0 - 2	Draw by 3-fold repetition
1 - 0 - 2	White wins by check mate
1 - 1 - 2	White wins by check mate
1 - 2 - 2	Black wins by check mate
1 - 3 - 2	White wins by check mate
1 - 4 - 2	Black wins by check mate
2 - 4 - 2	Black wins by check mate
2 - 5 - 2	Black wins by check mate
2 - 6 - 2	White wins by check mate

Once again lots of limitations:
* the time control is hardcoded (100ms)
* the UCI options are left to their default value (although the machinery to load the available ones and modify them is there in the code, just not plugged to anything)

But before I write more code, which will inevitably turn into a pile of crap, that I'll have to rewrite, I need to think about the architecture:
* there would be two kinds of tournaments (Round-Robin and Gauntlet)
* there would be a given number of concurrent games, each running in a thread (communicating to two child processes).
* there would have to be some kind of scheduler, that distributes the games to be played across the threads. The more I think of it the more complicated it seems... For example, I don't want to systematically kill/spawn processes (only terminate a process when it has finished its gauntlet, and an RR is seen as a series of gauntlets here).

So I'm wondering how to architecture (in terms of classes and modules) all this, into a soundly designed and extensible framework.

Ideas welcome