Android tournament with longer time controls

[abik]

The list of UCI and XBoard engines for Android is getting longer, and I am pondering about a more serious tournament at longer time control in Chess for Android. As a first step, I tested all available Android binaries. The list is shown below. A "X" indicates that the engine has trouble running in Chess for Android (if you are the engine author and are interested in fixing this, please contact me for details).

Next, I probably do a few quick tournaments to divide the engine in groups by strength, followed by tournaments at longer time control. Please feel free to contact me if you are interested in participating, but don't know how to compile your engine for Android (ARM) or if you have good ideas about the format of this tournament. I am not really an experienced tournament manager, but merely do this for fun to get an idea of the relative strength of these engines.

Code: Select all

apilchess-106-ja           X    U. Lorenz, C. Donninger
bikjump1.8                      Aart J.C. Bik
bikjump2.1                      Aart J.C. Bik
cassandre-026-ja           X    Jean-Francois Romang, et al.
chess for android               Aart J.C. Bik
critter-12-arm                  Richard Vida
cyrano-061-b17-ja          X    Harald Johnsen
daydreamer-075-ja               Aaron Becker
diablo-051b-ja                  Marcus Prewarski
diablo-501-ja                   Marcus Prewarski
dirty-android              X    Andres Valverde
doublecheck1.1                  Lucas Braesch
doublecheck1.2                  Lucas Braesch    
doublecheck1.3.1                Lucas Braesch
fairymax48q                     H.G. Muller
garbochess3                     Gary Linscott
garbochess-3-ja                 Gary Linscott
gaviota0.80                     Miguel A. Ballicora
gaviota0.82beta                 Miguel A. Ballicora
gaviota0.83.0.3                 Miguel A. Ballicora
gaviota-android-v0.83           Miguel A. Ballicora
gaviota-android-v0.84           Miguel A. Ballicora
gnuchess-5.07.153.3b            Michel van den Bergh, et al.
gnuchess-5.07.170b.5b           Michel van den Bergh, et al.
gnuchess-5.07.173b-android      Michel van den Bergh, et al.
greko8.0                        Vladimir Medvedev
IvanHoe-999947c                 Yakov Petrovich Golyadkin, et al.
IvanHoe-999947c-robbo           Yakov Petrovich Golyadkin, et al.
jazz-444-ja                     Evert Glebbeek
jazz-uci-501-ja                 Evert Glebbeek
komodo-13-ja                    Don Dailey, Larry Kaufman
komodo201-ab                    Don Dailey, Larry Kaufman
komodo203-ab                    Don Dailey, Larry Kaufman
komodo-203-ja                   Don Dailey, Larry Kaufman
komodo2-ab                      Don Dailey, Larry Kaufman
komodo3-ab                      Don Dailey, Larry Kaufman
pepito-159-ja                   Carlos del Cacho
redqueen095                     Ben-Hur Carlos Vieira Langoni Jr.
redqueen098                     Ben-Hur Carlos Vieira Langoni Jr.
redqueen-098-ja                 Ben-Hur Carlos Vieira Langoni Jr.
robbolito0085e4l                Yakov Petrovich Golyadkin, et al
robbolito0085g3l                Yakov Petrovich Golyadkin, et al
rocinante-101-ja           X    Antonio Torrecillas
rotor0.6a                       Jan Brouwer
rotor-04-ja                     Jan Brouwer
simplex-097-ja                  Antonio Torrecillas
sissa-20-ja                     Christophe J. Mandin
stockfish_crab1                 modified by Adam Kleng
stockfish1.8                    Tord Romstad, Marco Costalba, Joona Kiiski
stockfish1.9                    Tord Romstad, Marco Costalba, Joona Kiiski
stockfish2.0                    Tord Romstad, Marco Costalba, Joona Kiiski
stockfish2.1                    Tord Romstad, Marco Costalba, Joona Kiiski
stockfish-21-android-ja         Tord Romstad, Marco Costalba, Joona Kiiski
stockfish-211-android-ja        Tord Romstad, Marco Costalba, Joona Kiiski
suff-03a-ja                X    Clemens Pruell
sungorous-14-ja            X    Pablo Vazquez
toga2                           Thomas Gaksch, Fabien Letouzey
umax48w                         H.G. Muller
zct032500                       Zach Wegner
zeta-dva-0202-ja           X    Srdja Matovic

[Jim Ablett]

Hi Aart,

I tried compiling/running a few winboard engines. I couldn't get any to run
unfortunately. You can add these winboard engines to the list of currently not working >

scorpio 2.7..........X..........Daniel Shawul
bruja 1.91...........X..........Dan Honeycutt
phalanx xxii........X..........Dusan Dobes
crafty 23.4..........X..........Robert Hyatt
amundsen 0.80...X..........John Bergbom
sloppy 0.22.........X..........Ilari Pihlajisto
kiwi 0.6d............X...........Alessandro Scotti
prophet 2.0b1.....X...........James Swafford

Jim.

[abik]

I tried compiling/running a few winboard engines. I couldn't get any to run
unfortunately. You can add these winboard engines to the list of currently not working[

Thanks Jim. XBoard support in CfA is still in "alpha", so with so many engines not working, it is more likely that the GUI is at fault
I will have a look later....

[abik]

I ran tournaments between different versions of the same engine to determine which one would be their representative. Below I show the results of a fast round-robin tournaments using Chess for Android on a Nexus One, Nexus S, and Galaxy Tab with, where applicable, 8MB hash table and 1MB tablebases cache (Nalimov, Gaviota, Scorpio, and Robbobases reside on SD card). Each version played each other version from both sides of the Nunn opening suite.

The number one engine will play in the longer time control tournament (which is not always the latest version; engine authors let me know if you prefer otherwise since some results were pretty close). Also, Pablo Vazquez and Jim Ablett kindly worked together to fix the issue in the sungorus. It now runs correctly and will participate in the tournament.

Code: Select all

1   BikJump v2.1P       10.5/20
2   BikJump v1.8         9.5/20

1   Diablo 0.5.1b JA    17.5/20
2   Diablo 0.5.1         2.5/20
                    
1   DoubleCheck 1.3     13.5/20
2   DoubleCheck 1.2      6.5/20

1   gaviota v0.84       47.5/80
2   gaviota v0.83.0.3   45.0/80
3   gaviota v0.83       41.0/80
4   gaviota v0.80.0.107 37.5/80
5   gaviota v0.82-beta2 29.0/80

1   GNU Chess 5.07.153.3b 20.5/40
2   GNU Chess 5.07.170.7b 20.0/40
3   GNU Chess 5.07.173b   19.5/40
                                    
1   IvanHoe-Beta v999947c-rob 10.0/20
2   IvanHoe-Beta v999947c     10.0/20
                             
1   Jazz v5.01 JA       14.0/20
2   Jazz v444 JA         6.0/20

1   Komodo32 2AB       60.0/100
2   Komodo32 2.03 AB   59.0/100
3   Komodo32 2.01 AB   58.5/100
4   Komodo32 2.03 JA   58.0/100
5   Komodo32 3 AB      34.5/100
6   Komodo32 1.3 JA    30.0/100
                    
1   RedQueen 0.9.8 JA  24.5/40
2   RedQueen 0.9.8     19.0/40
3   RedQueen 0.9.5     16.5/40
                            
1   RobboLito 0.085g3l 10.5/20
2   RobboLito 0.085e4l  9.5/20
                
1   Rotor 0.6a         16.0/20
2   Rotor 0.4           4.0/20

1   Stockfish 2.1.1 JA 77.0/120
2   Stockfish 2.1      68.5/120
3   Stockfish 2.1 JA   65.5/120
4   Stockfish 1.8      62.0/120
5   Stockfish 2.0      60.0/120
6   Stockfish 1.9      59.0/120
7   Crab 1.0 beta      28.0/120

[abik]

I tried compiling/running a few winboard engines

Thanks again Jim. I had a quick look, and got some of these engines working with relatively little GUI improvements. One binary simply exits, however, while some others do not seem to adhere very strictly to the standard (my GUI is still rather rigid for XBoard, I suspect most established XBoard GUIs are more forgiving). I will try to use at least a few of this list in the upcoming tournament though, to make it more interesting.

[abik]

The following XBoard and UCI engines now pass a quick test tournament from both sides of the first Nunn position and are ready to enter the Chess for Android tournament too. Next up: qualifier tournament to split the engines into smaller groups for the longer time control matches.

New entries:

Code: Select all

crafty 23.4                  Robert Hyatt 
scorpio 2.7                  Daniel Shawul 
sloppy 0.22                  Ilari Pihlajisto 
tscp1.81                     Tom Kerrigan
sungorous-14-ja              Pablo Vazquez

Cross-table of test tournament:

Code: Select all

                       1  2  3  4  5  6  
1   scorpio            ** 01 11 11 11 11   9.0/10
2   crafty             10 ** ½1 1½ 11 11   8.0/10
3   sloppy             00 ½0 ** ½1 11 11   6.0/10
4   Sungorus 1.4 JA    00 0½ ½0 ** 11 11   5.0/10
5   tscp181            00 00 00 00 ** 11   2.0/10
6   Chess for Android  00 00 00 00 00 **   0.0/10

[abik]

Jim,
I made various other improvements and added few of the idioms described in Tim Mann and H.G. Muller's detailed document (for example, recognizing an old idiom like "5. ... f8e7" instead of "move f8e7" and accepting SAN instead of coordinate algebraic only). Currently, the status compared to what you reported earlier is as follows (expect an update to Chess for Android soon!).

For the remainder 'X's, I believe the engine is at fault.

Code: Select all

scorpio 2.7.....................Daniel Shawul 
bruja 1.91...........X..........Dan Honeycutt 
phalanx xxii....................Dusan Dobes 
crafty 23.4.....................Robert Hyatt 
amundsen 0.80........X..........John Bergbom 
sloppy 0.22.....................Ilari Pihlajisto 
kiwi 0.6d.......................Alessandro Scotti 
prophet 2.0b1........X..........James Swafford

[abik]

Below the results of a qualification tournament on Chess for Android from both sides of the first Nunn position at one-second per move on a Nexus One (1GHz Snapdragon). Where applicable, UCI engines were given an 8MB hash table and 1MB endgame tablebases cache (the complete 3- and 4-piece Nalimov, Gaviota, Scorpio, and Robbobases endgame tablebases reside on SD card). XBoard engines play at their default settings. Since the Nexus One is a single-core device, "pondering" was disabled

The qualification results are used to split the engines into separate groups. Engines in each group will play each other at longer time controls, starting with the lowest group. The number one in each group will promote to the next group.

All games can be downloaded from the tournament page.

Code: Select all

                                    1  2  3  4  5  6  7  8  9  0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5  6  7  8  
1   Critter 1.2 32-bit              ** ½0 ½1 10 11 11 11 11 11 1½ 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11  50.5/54
2   Stockfish 2.1.1 JA              ½1 ** 01 0½ 11 11 11 01 11 11 ½1 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11  49.5/54
3   RobboLito 0.085g3l x86          ½0 10 ** 01 10 11 11 11 11 11 01 11 11 11 ½1 11 11 11 11 11 11 11 11 11 11 11 11 11  48.0/54
4   Komodo32 2AB                    01 1½ 10 ** 01 ½½ 11 10 11 11 11 11 01 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11  47.5/54
5   IvanHoe-Beta version 999947c-r  00 00 01 10 ** 11 11 1½ 10 11 11 ½1 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11  46.0/54
6   crafty                          00 00 00 ½½ 00 ** 01 11 01 ½1 ½½ ½1 11 ½1 ½1 ½1 11 1½ 11 11 11 11 11 11 11 11 11 11  39.0/54
7   Toga II 1.4.1SE                 00 00 00 00 00 10 ** 01 11 11 1½ ½1 1½ 11 01 ½½ ½1 11 11 11 11 11 11 11 11 11 11 11  38.0/54
8   scorpio                         00 10 00 01 0½ 00 10 ** 0½ ½½ 11 01 ½1 1½ 11 10 11 10 01 11 11 11 11 ½1 11 11 11 11  35.5/54
9   GarboChess3                     00 00 00 00 01 10 00 1½ ** 00 01 10 11 01 ½½ 11 11 1½ 11 ½1 11 11 11 11 11 11 11 11  34.5/54
10  Rotor 0.6a                      0½ 00 00 00 00 ½0 00 ½½ 11 ** ½½ 01 11 10 1½ ½½ 01 10 11 11 11 11 11 11 11 11 11 11  33.5/54
11  GNU Chess 5.07.153.3b-32        00 ½0 10 00 00 ½½ 0½ 00 10 ½½ ** 10 11 01 ½½ 11 ½½ 1½ ½1 11 11 1½ 1½ 11 11 11 11 11  33.0/54
12  Pepito v1.59                    00 00 00 00 ½0 ½0 ½0 10 01 10 01 ** ½0 ½½ 11 ½0 ½0 1½ 11 11 01 11 11 1½ 11 11 11 11  30.0/54
13  Daydreamer 1.75 JA              00 00 00 10 00 00 0½ ½0 00 00 00 ½1 ** 11 10 11 11 ½0 10 11 ½1 11 11 11 11 11 11 11  29.5/54
14  gaviota v0.84                   00 00 00 00 00 ½0 00 0½ 10 01 10 ½½ 00 ** ½½ 10 1½ 11 1½ 01 11 11 11 11 11 11 11 11  29.0/54
15  Diablo 0.5.1b JA                00 00 ½0 00 00 ½0 10 00 ½½ 0½ ½½ 00 01 ½½ ** 10 ½1 ½½ 0½ 11 11 11 11 11 11 11 11 11  28.5/54
16  sloppy                          00 00 00 00 00 ½0 ½½ 01 00 ½½ 00 ½1 00 01 01 ** ½0 ½1 11 11 11 1½ 11 11 11 11 11 11  28.5/54
17  RedQueen 0.9.8 JA               00 00 00 00 00 00 ½0 00 00 10 ½½ ½1 00 0½ ½0 ½1 ** 11 01 11 11 1½ 11 11 11 11 11 11  27.0/54
18  Sungorus 1.4 JA                 00 00 00 00 00 0½ 00 01 0½ 01 0½ 0½ ½1 00 ½½ ½0 00 ** 11 11 1½ 1½ 11 11 11 11 11 11  26.0/54
19  GreKo 8.0                       00 00 00 00 00 00 00 10 00 00 ½0 00 01 0½ 1½ 00 10 00 ** ½1 11 11 10 11 11 11 11 11  22.0/54
20  DoubleCheck 1.3                 00 00 00 00 00 00 00 00 ½0 00 00 00 00 10 00 00 00 00 ½0 ** 01 01 1½ 11 11 11 11 11  15.5/54
21  Jazz v5.01 JA                   00 00 00 00 00 00 00 00 00 00 00 10 ½0 00 00 00 00 0½ 00 10 ** 11 00 1½ 11 11 11 11  14.5/54
22  BikJump v2.1P                   00 00 00 00 00 00 00 00 00 00 0½ 00 00 00 00 0½ 0½ 0½ 00 10 00 ** 11 1½ 11 ½1 11 11  14.0/54
23  ZCT-0.3.2500                    00 00 00 00 00 00 00 00 00 00 0½ 00 00 00 00 00 00 00 01 0½ 11 00 ** 11 11 11 11 11  14.0/54
24  tscp181                         00 00 00 00 00 00 00 ½0 00 00 00 0½ 00 00 00 00 00 00 00 00 0½ 0½ 00 ** 1½ 11 11 11   9.5/54
25  umax48w                         00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0½ ** ½1 10 11   5.0/54
26  fairymax48q                     00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ½0 00 00 ½0 ** 1½ 11   4.5/54
27  Chess for Android               00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 0½ ** 11   3.5/54
28  Simplex 0.9.7 rev 180           00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 **   0.0/54

[abik]

Games in group F are running. At 30 seconds-per-move for both sides of all Nunn positions, I suspect running the games for each group of five will take several days at least.

Group A
Critter 1.2
Stockfish 2.1.1 JA
RobboLito 0.085g3l x86
Komodo32 2AB
+ winner B

Group B
IvanHoe-Beta version 999947c
crafty
Toga II 1.4.1SE
scorpio
+ winner C

Group C
GarboChess3
Rotor 0.6a
GNU Chess 5.07.153.3b
Pepito v1.59
+ winner D

Group D
Daydreamer 1.75 JA
gaviota v0.84
Diablo 0.5.1b JA
sloppy
+ winner E

Group E
RedQueen 0.9.8 JA
Sungorus 1.4 JA
GreKo 8.0
DoubleCheck 1.3
+ winner F

Group F
Jazz v5.01 JA
BikJump v2.1P
ZCT-0.3.2500
tscp181
umax48w

[Michel]

Hey, it seems GNU Chess does quite well on android given its ancient origin.

On the negative side: the improvements in my two latest released versions
seem to have little (if any) positive effect on android....

Thanks for the tournament.