Chessultant - UCI chess-cluster-engine

[Zagalo]

Dear all,

i have seen a few people in the "Rybka 4 Update"-Thread (may)being interested
into my personal chess-cluster-software which i have developed over the last
few years for my own purposes. It didn't even had a name till today (used to
be just chess[.exe]), so i decided to name it "Chesssultent" starting today.

In this thread i want to show you the current capabilities and limitations of
Chesssultent, want to evaluate whether there is a bigger interest in a public
release of a freely available version of it (which would justify the further
work needed) and also want to collect any ideas & suggestions you might have.

Chesssultent is a chess-cluster-software which is using currently & publicly
available single-PC x86-chess-engines to evaluate a chess position on a (in
best case symmetric) hardware-cluster. Chesssultent is capable of using any
UCI-conform engine for its analysis and is itself also a UCI-confirm engine
coming with its own little GUI to control the calculation and providing in-
depth analysis & logging of the current postion. It's written in plain C/C++
(the GUI being based on the Qt-library), being carefully handoptimized and
can be configured with an options-file and/or parameters and can run under
the Micro$oft Windows Family or any Linux-distribution using a KDE-frontend.
It's currently based on & optimized to my hardware-cluster (see below for
detailed specifications) consisting of a single master (M) and 16 slaves (S).

Chesssultent can provide several modes of calculation:

- Width: Two phases:
o First Phase: (M) runs a master-engine with Multi-PV-analysis,
the best 16 moves are moved for further Multi-PV-analysis on the (S).
If a new entry reaches Top16 on (M) and holds there for a threshold
(adjustable depth/time), the (S) analysing the weakest move is stopped
and the new Top16-entry is moved to this (S). The (M) can also run another
engine in parallel on limited cores to solve tactical/win-finding problems.
o Second Phase: out of this +16x16-move-matrix following possibilities arise:
a) the overall best 16 moves are choosen for further analysis on (S)
b) the 4 best B-answers to any of the 4 best A-moves are choosen for
for further analysis on (S)
c) the best 8 B-answers to the best A-move, the best 4 B-answers to
the second best A-move and the 2 best answers to the third and forth
best (or one forth/fifth) A-moves are choosen for further analysis on (S)
d) there is already a little intelligence implemented which does categorize
the position tactical/positional, open/closed and winning/drawish/losing,
considering available time and the number of moves played, and can choose
out of a)-c) by itself and maybe also build a more intelligent custom
search-pattern in future (but this topic is a whole Ph.D. by itself ).
The master does analyze the non-Top16 of the first Phase meanwhile and
does move upmovers to a (S) for further analysis. It can also analyze the
current best move off all (S) or continue its calculation of the first Phase.
Finally (M) takes the best move which reached a reasonable depth in analysis,
calculates on while waiting for black's move and then starts the next iteration.

- Depth: Again (M) runs a Multi-PV-analysis. Optionally adjustable,
(M) runs further four engines to analyze the best 4 A-moves out of
this analysis and moves the best 4x4 AxB-moves to the (S) for further analysis.
It's also possible that 4 (S) run the B-move-analysis instead of (M),
then the best 4x3 moves are moved to the (S) or any customized number.
Also the same intelligence as described under "-Width" above can be used.
Finally (M) takes the best move which reached a reasonable depth in analysis.

- Quick: Again (M) runs a Multi-PV-analysis, 8 (S) calculate the best B-moves to
this A-moves, the best 4 (or a custom pattern) are analyzed in parallel on 4 (S),
and these best C-moves are analyzed on 2 (S) and the best D-moves on another 2 (S).
Finally (M) takes the best move which reached a reasonable depth in analysis.
- Aggressive: Same as Quick, takes two moves at once and therefore doubles depth
for sake of width.

- Engine-Group: Again (M) runs a Multi-PV-analysis.
o First Phase: the 8/4/2 best moves are moved to a (S)-group consisting of 2/4/8
strong engines (which have nearly equally strength but a different character and
reach a similiar analysis-depth a time in best case) for further analysis.
o Second Phase: There are two different possibilites now:
a) The leading engine calculates the analysis of the created 16-moves-matrix
out of the first Phase or an adjustable custom pattern on the (S).
b) The groups analyze the best 4 moves out of the 16-moves-matrix each.
Finally (M) takes the best move which reached a reasonable depth in analysis,
in case of b) it may choose to most favored move or a custom pattern.

- Nalimov: In a Nalimov(currently: 6)-position only (M) runs.

Chesssultent is capable of saving&using persistent hash-files, providing the (S)
with the best-suited one out of the 16x16x16-matrix it has on file, which
consums pretty exactly a terabyte for each 256 MB hash-size on harddisk.
In case available time is not infinite as in UCI analysis-mode, my time-per-
move algorithms are based on a degressive schedule involing the balance of
the current position, it spends most time in early moves and drawn positions.

Chesssultent as of today has the following known limitations:
- It is a optimized for a symmetrical architecture, and while it would be
pretty easy to scale it upwards (i'm not doing this soon, my electricy-
bill has already reached 10k ), downwards will be a more difficult task.
I would even say that the number of (S) should be a magnitude of 2 to
make things easier, another number and/or unsymmetrical architectures
would need far more balancing and intelligence in the analysis of results.
- It is best suited to analyzing. It can play games with long time-frames,
it must be further optimized for blitz/rapid and there isn't to much sense
of using it for (kiddy-)bullet, my interest is in correct chess anyways.
Anyways there is room for other (experimental) time-per-move algorithms.
- The intelligence of creating (S)-patterns depending on the current
position for sure has a Ph.-D.-room to improve.
- It needs a huge network-bandwith, especially for the persistant hash-files.
Maybe i will implement some kind of compression there.
- When to stop a (S) for another calculation is also a science for itself.
- The GUI suites my needs, it may not suite yours. But you can use it with
your favorite GUI which can show the best (PV-)lines, everything else is
in the logs and you may write scripts to handle that for further analysis.
- Experience with the modes and all the parameters is of course limited,
there may be better ones and a lot room for further improvement.
- Thousands other limitations that i can't think of at the moment.

Anyways, it plays pretty good chess and beats any single engine mostly with ease.
It already has analyzed several openings to win against itself, but as you know,
this often only holds until it is refuted by a bigger approach - that is nature.

My cluster currently consists of the following components:
- 1x Master: (Brandnew) 2x Intel Xeon DP X5680 (12 Cores @ 4,0GHz) running on a
(brandnew&lovely) EVGA Classified SR-2 with Intel Server NIC and 24GB Ram, the
most beautiful piece of (affordable) x86-workstation/server-hardware ever seen.
- 16x Slaves: (Mature) Intel Xeon UP W3520 (4 Cores @ 4,0GHz) running on a
Asus P6T Deluxe V2 with Intel NIC and 6GB Ram
- 1x Switch: (Young) Cisco Catalyst 2960G-24TC

Regarding my person:
I'm a self-employed participant of financial markets interested in chess-analysis,
especially analysing interesting positions and openings, developing opening-systems
while refuting others and finding the "perfect game". I'm currently providing my
analysis to my favorite two GM's and one IM, and myself having only rarely interest
and time for over_the_board-play and even less for point-hunting on chess-servers.

Please share what you think about Chesssultent with me, your ideas & suggestions
to further improve it and bring it to a useable public release, and also what kind
of cluster-specifications you have at your hands and what i may can do for you.

Thanks for all your time,

Alex

http://rybkaforum.net/cgi-bin/rybkaforu ... ?tid=16328