Chess variant tournament: Spartan Chess

[enhorning]

Cross-table after 5 cycles out of 6:

Code: Select all

Cross table, sorted by score percentage, Buchholz, SB

                              Sangg Nebiy Spart Sjaak Leoni Pair- Fairy Catal
 1. Sangga v1.0               ##### 10011 01111 =1111 1101= 1=111 11111 11111
                              ##### 11101 11011 11111 11111 11111 11111 11111   89%  62.5 (2175.0, 1841.3)

 2. NebiyuChess_1.43          01100 ##### 01101 11111 11=11 10111 11111 11111
                              00010 ##### 10=11 11111 11111 11111 01111 11111   81%  57.0 (2230.0, 1594.5)

 3. Spartacus 0.24            10000 10010 ##### 00=00 01101 11111 11111 11111
                              00100 01=00 ##### 1=000 11001 11111 11111 11011   61%  42.5 (2375.0, 1042.0)

 4. Sjaak $Rev: 480 $         =0000 00000 11=11 ##### 0111= 11101 1111= 11111
                              00000 00000 0=111 ##### 0=011 1=101 =111= 11111   58%  40.5 (2395.0, 949.5)

 5. Leonidas 8.3              0010= 00=00 10010 1000= ##### 101=1 11111 10101
                              00000 00000 00110 1=100 ##### 1=00= =111= 11110   45%  31.5 (2485.0, 771.5)

 6. Pair-o-Max 4.8S           0=000 01000 00000 00010 010=0 ##### 0==11 1=0=1
                              00000 00000 00000 0=010 0=11= ##### 10=01 11=10   29%  20.5 (2595.0, 484.0)

 7. Fairy-Max 4.8Rt           00000 00000 00000 0000= 00000 1==00 ##### 1101=
                              00000 10000 00000 =000= =000= 01=10 ##### 0001=   19%  13.0 (2670.0, 304.0)

 8. Catalyst                  00000 00000 00000 00000 01010 0=1=0 0010= #####
                              00000 00000 00100 00000 00001 00=01 1110= #####   18%  12.5 (2675.0, 273.8)

[enhorning]

Final cross-table:

Code: Select all

Cross table, sorted by score percentage, Buchholz, SB

                              Sangga Nebiyu Sparta Sjaak  Leonid Pair-o Cataly Fairy-
 1. Sangga v1.0               ###### 100111 011111 =11111 1101=1 1=1111 111111 111111
                              ###### 110101 101110 111111 111111 111111 111111 111111   89%  74.5 (3138.0, 2613.0)

 2. NebiyuChess_1.43          011000 ###### 011011 111111 11=111 101111 111111 111110
                              001010 ###### 0=1111 111110 111111 111111 111111 111111   82%  69.0 (3204.0, 2320.0)

 3. Spartacus 0.24            100000 100100 ###### 00=001 011011 111111 111111 111111
                              010001 1=0000 ###### =00010 100111 111111 101111 11111=   62%  52.0 (3408.0, 1548.5)

 4. Sjaak $Rev: 480 $         =00000 000000 11=110 ###### 0111=0 111011 111111 1111==
                              000000 000001 =11101 ###### =01101 =10100 111111 111=11   57%  47.5 (3462.0, 1367.5)

 5. Leonidas 8.3              0010=0 00=000 100100 1000=1 ###### 101=11 101011 11111=
                              000000 000000 011000 =10010 ###### =00=11 111011 111=1=   45%  38.0 (3576.0, 1072.5)

 6. Pair-o-Max 4.8S           0=0000 010000 000000 000100 010=00 ###### 1=0=11 0==111
                              000000 000000 000000 =01011 =11=00 ###### 1=1001 0=0110   29%  24.5 (3738.0, 704.8)

 7. Catalyst                  000000 000000 000000 000000 010100 0=1=00 ###### 0010=1
                              000000 000000 010000 000000 000100 0=0110 ###### 110=11   18%  15.5 (3846.0, 381.3)

 8. Fairy-Max 4.8Rt           000000 000001 000000 0000== 00000= 1==000 1101=0 ######
                              000000 000000 00000= 000=00 000=0= 1=1001 001=00 ######   18%  15.0 (3852.0, 435.5)

Congratulations to Sangga for its impressive performance!

Running the pgn-file through bayes-elo gives this:

Code: Select all

ResultSet-EloRating>ratings
Rank Name                Elo    +    - games score oppo. draws
   1 Sangga v1.0         364  140  140    84   89%   -52    4%
   2 NebiyuChess_1.43    292  127  127    84   82%   -42    2%
   3 Spartacus 0.24       94  109  109    84   62%   -13    5%
   4 Sjaak $Rev: 480 $    44  104  104    84   57%    -6   11%
   5 Leonidas 8.3        -48  103  103    84   45%     7   12%
   6 Pair-o-Max 4.8S    -184  107  107    84   29%    26   13%
   7 Fairy-Max 4.8Rt    -272  113  113    84   18%    39   14%
   8 Catalyst           -288  119  119    84   18%    41    6%
ResultSet-EloRating>los
                   Sa Ne Sp Sj Le Pa Fa Ca
Sangga v1.0           86 99 99 99 99100100
NebiyuChess_1.43   13    99 99 99 99100100
Spartacus 0.24      0  0    82 99 99 99 99
Sjaak $Rev: 480 $   0  0 17    96 99 99 99
Leonidas 8.3        0  0  0  3    99 99 99
Pair-o-Max 4.8S     0  0  0  0  0    94 96
Fairy-Max 4.8Rt     0  0  0  0  0  5    60
Catalyst            0  0  0  0  0  3 39

So Pair-o-Max is better than Fairy-Max with 94% confidence, and Sjaak is better than Leonidas with 96% confidence, according to these results.

Final tally was 152 wins for the Fide side, 156 wins for the Spartans, and 28 draws, so looking balanced. As usual, if anybody wants the full pgn-file, either pm me or email me at enhorning@gmail.com .

Next tournament, in a couple of weeks, give or take, will be Shatranj, for which I preliminarily have the following participants: Sha-Max, Nebiyu, Pulsar, Sjaak, Dabbaba, Deuterium derivative. Feel free to point out any other programs that can play Shatranj to me!

[Evert]

Final cross-table:

Code: Select all

Cross table, sorted by score percentage, Buchholz, SB

                              Sangga Nebiyu Sparta Sjaak  Leonid Pair-o Cataly Fairy-
 1. Sangga v1.0               ###### 100111 011111 =11111 1101=1 1=1111 111111 111111
                              ###### 110101 101110 111111 111111 111111 111111 111111   89%  74.5 (3138.0, 2613.0)

 2. NebiyuChess_1.43          011000 ###### 011011 111111 11=111 101111 111111 111110
                              001010 ###### 0=1111 111110 111111 111111 111111 111111   82%  69.0 (3204.0, 2320.0)

 3. Spartacus 0.24            100000 100100 ###### 00=001 011011 111111 111111 111111
                              010001 1=0000 ###### =00010 100111 111111 101111 11111=   62%  52.0 (3408.0, 1548.5)

 4. Sjaak $Rev: 480 $         =00000 000000 11=110 ###### 0111=0 111011 111111 1111==
                              000000 000001 =11101 ###### =01101 =10100 111111 111=11   57%  47.5 (3462.0, 1367.5)

 5. Leonidas 8.3              0010=0 00=000 100100 1000=1 ###### 101=11 101011 11111=
                              000000 000000 011000 =10010 ###### =00=11 111011 111=1=   45%  38.0 (3576.0, 1072.5)

 6. Pair-o-Max 4.8S           0=0000 010000 000000 000100 010=00 ###### 1=0=11 0==111
                              000000 000000 000000 =01011 =11=00 ###### 1=1001 0=0110   29%  24.5 (3738.0, 704.8)

 7. Catalyst                  000000 000000 000000 000000 010100 0=1=00 ###### 0010=1
                              000000 000000 010000 000000 000100 0=0110 ###### 110=11   18%  15.5 (3846.0, 381.3)

 8. Fairy-Max 4.8Rt           000000 000001 000000 0000== 00000= 1==000 1101=0 ######
                              000000 000000 00000= 000=00 000=0= 1=1001 001=00 ######   18%  15.0 (3852.0, 435.5)

Congratulations to Sangga for its impressive performance!

Quite happy with how Sjaak came out of this. Interesting that it scored so well against Spartacus, but worse against Nebiyu and Sangga. Bit disappointed about Leonidas, but I think I found (and fixed) the problem. I'll need to test it at long time controls (as opposed to "longer" time controls) to be certain though. I'll make sure it's in good shape for the next Spartan match (whenever that is).

Next tournament, in a couple of weeks, give or take, will be Shatranj, for which I preliminarily have the following participants: Sha-Max, Nebiyu, Pulsar, Sjaak, Dabbaba, Deuterium derivative. Feel free to point out any other programs that can play Shatranj to me!

That'll be interesting. I never particularly optimised Sjaak for playing it so the evaluation is probably going to be wildly mis-tuned. There are some evaluation components that I know are important that I haven't implemented. It'll be interesting.

Did you say you were going to do a Seirawan tournament at some point as well?

[hgm]

Thanks very much for running this tourney! It was very interesting, and in addition has led to discovery of quite a few bugs and imperfections in Fairy-Max and WinBoard. The version of WinBoard 4.7.0 that will be eventually released will have no problem running engines like Oberon, which do not send feature done=1, as a result of this. (It will just take 10 sec longer to start them up.) Fairy-Max will no longer claim a Persian win if there is a Hoplite on 2nd rank.

It is interesting to compare this result with that of the Thermopylae Marathon event I conducted two years ago:

Code: Select all

Cross table, sorted by score percentage, Buchholz, SB

                              Spa Neb Sja Obe Fai Che Cat
 1. Spartacus 0.23            ### =10 011 10= =11 111 111
                              ### 111 111 1   010 1   0     75%  22.5 (391.0, 289.8)

 2. Nebiyu 1.1                =01 ### 11= 111 111 0=1 111
                              000 ### 0=1 0   111 0   1     67%  20.0 (406.0, 243.5)

 3. Sjaak 92                  100 00= ### 101 1=1 001 110
                              000 1=0 ### 1   011 1   1     52%  15.5 (433.0, 188.5)

 4. Oberon                    01= 000 010 ### 01= 10= 111
                              0   1   0   ### 0   1   1     48%  11.5 (338.0, 131.5)

 5. Fairy-Max  4.8R           =00 000 0=0 10= ### 11= 111
                              101 000 100 1   ### 1   0     43%  13.0 (448.0, 152.5)

 6. ChessV (Spartan)          000 1=0 110 01= 00= ### 100
                              0   1   0   0   0   ### 0     31%   7.5 (354.0, 110.8)

 7. Catalyst 3                000 000 001 000 000 011 ###
                              1   0   0   0   1   1   ###   25%   6.0 (360.0,  73.5)

The Spartacus version participating in both tourneys is practically the same. (0.24 is most likely a replacement of 0.23 playing the final games of the Thermopylae Marathon, judging by the date of the .exe.) It seems that the other engines progressed a lot, and that Sangga is a great addition. The Thermopylae Marathon was conducted on a 32-bit PC, and I understand this tourney was 64-bit. It seems Sjaak benefits a lot from this, and I don't know how that is with Nebiyu (which is a much newer version anyway).

Indeed Sjaak seems to do unusually well against Spartacus, in the direct confrontation. I wonder if this can be explained from tactical vs strategic abilities. It would be interesting to see the games!

[enhorning]

Did you say you were going to do a Seirawan tournament at some point as well?

Seirawan would be an option if I could get enough programs... I'm happy to run tournaments for any variants (apart from 960) which has enough programs and which can be run through Winboard. Looking at my notes, Capablanca, Courier, Makruk, Nightrider and Seirawan Chess are the ones remaining that I have at least 3 programs for. (Unless I get drastic and branch out to Shogi / XiangQi... don't know how many WB-compatible engines there exist for those - and at least for Shogi, WB notation is highly confusing as it uses Fide Chess coordinates instead of traditional Shogi coordinates!)

[hgm]

Well, I always considered it the Shogi coordinates to be confusing... Shogi engines that I managed to run under WinBoard are GPS Shogi, Blunder, Bonanza, Spear, Laramie, PetitShogi, TJshogi, Lesserkai and of course my own Shokidoki. For mini-Shogi there is only Shokidoki, TJshogi5x5 and PetitShogi. Not all engines support all time-control modes, though, but they can usually be made to approximate incremental mode.

For Xiangqi there are many dozens of engines that can run under WinBoard.

I am in possession of a super-strong commercial Seirawan engine, but it was given to me in confidence (to see if we could get it to run under WinBoard through UCI2WB), so I am not at liberty to even share it as a private engine. I can contact the author, though, to ask if he would be willing to enter it in any tourney.

Unfortunately Spartacus does not play Seirawan Chess yet. I had a severe problem encoding the gating moves there, because there are so many. (Spartacus' system for encoding non-standard moves is by using an off-board to-square of its 24x10 board array. But quite a few of these codes are already in use for encoding promotions, castlings, e.p. captures and double-pushes. I could make use of the fact that the board is only 8x8 in Seirawan Chess, so that the 'i'- to 'p'-file are also available for special encodings (the others use only 'q'- to 'x'-file), but even then there are lots of possible moves, which each have to combine with two possible gatings (E or H).

Spartacus also does not play Shatranj. Originally I intended to implement that, but in a test match between Fairy-Max and Pulsar I broadcasted Shatraj proved such a boring game that I never bothered...

ChessV plays Shatranj, though. (There is an unreleased version of ChessV that plays Spartan, but its author has disappeared from the radar, so I cannot ask his permission to share it.)

[Evert]

Thanks very much for running this tourney! It was very interesting, and in addition has led to discovery of quite a few bugs and imperfections in Fairy-Max and WinBoard. The version of WinBoard 4.7.0 that will be eventually released will have no problem running engines like Oberon, which do not send feature done=1, as a result of this. (It will just take 10 sec longer to start them up.)

I thought the problem was that it sent "done=1" rather than "feature done=1". That could be caught, right?
I know how you feel about working around engine bugs though.

The Spartacus version participating in both tourneys is practically the same. (0.24 is most likely a replacement of 0.23 playing the final games of the Thermopylae Marathon, judging by the date of the .exe.) It seems that the other engines progressed a lot, and that Sangga is a great addition. The Thermopylae Marathon was conducted on a 32-bit PC, and I understand this tourney was 64-bit. It seems Sjaak benefits a lot from this, and I don't know how that is with Nebiyu (which is a much newer version anyway).

Nebiyu isn't a bit-board program as far as I know, so I would expect very limited benefit from going to 64 bit. Sjaak is a bitboard program with (apparently) horrible 32 bit performance.
As I said, the gap will be even wider for large variants (which apparently don't work at all in 32 bit at the moment).

Indeed Sjaak seems to do unusually well against Spartacus, in the direct confrontation. I wonder if this can be explained from tactical vs strategic abilities. It would be interesting to see the games!

Yes indeed!
Sjaak is not particularly strong tactically, and I can't imagine how its generic evaluation can be strategically strong, but it must be doing something right.

[Evert]

Seirawan would be an option if I could get enough programs...

I would imagine that there'd be a few, but I can't actually name any (apart from Leonidas, which I should debug first), so that doesn't really help.

I'm happy to run tournaments for any variants (apart from 960) which has enough programs and which can be run through Winboard.

Awesome.
I always think variants don't get enough love on the forums (apart from 960, which as a variant isn't that interesting). I've been thinking on and off that there should be something like a "CCVRL", but I haven't got round to setting something up myself.
It's a nice niche though because there aren't that many variant engines around (compared to regular chess engines) and they aren't as overwhelmingly strong.

Looking at my notes, Capablanca, Courier, Makruk, Nightrider and Seirawan Chess are the ones remaining that I have at least 3 programs for.

Beware when playing Courier: at least Sjaak and WinBoard disagree on what the starting position should be. WinBoard uses the "all pieces on initial squares/back ranks" position, while Sjaak uses the position after the "compulsory" moves (edge pawns and queen pawns forward two places, queen forward two places, which would normally be an illegal move).
If you specify the FEN (either position will be fine, as long as all parties concerned agree on it) all is well.

(Unless I get drastic and branch out to Shogi / XiangQi... don't know how many WB-compatible engines there exist for those - and at least for Shogi, WB notation is highly confusing as it uses Fide Chess coordinates instead of traditional Shogi coordinates!)

Sjaak plays XiangQi.
Poorly, but it plays it. I had a dedicated XiangQi engine at some point, but I haven't touched it in years...

[Evert]

Unfortunately Spartacus does not play Seirawan Chess yet. I had a severe problem encoding the gating moves there, because there are so many. (Spartacus' system for encoding non-standard moves is by using an off-board to-square of its 24x10 board array. But quite a few of these codes are already in use for encoding promotions, castlings, e.p. captures and double-pushes. I could make use of the fact that the board is only 8x8 in Seirawan Chess, so that the 'i'- to 'p'-file are also available for special encodings (the others use only 'q'- to 'x'-file), but even then there are lots of possible moves, which each have to combine with two possible gatings (E or H).

I recently figured out a way that could possibly allow Sjaak to play Seirawan. Right now it would be impossible to encode castling moves that include a gate. It'd be a bit messy though and I'm reluctant to put myself through it.

An easy hack would be to have Sjaak never play gating moves from castling and just catch when the opponent does it at the top (similar to how Fairy-Max handles under promotions). Not sure what that would do to its playing style or strength though (but it'll still be better at it than it is now).

Spartacus also does not play Shatranj. Originally I intended to implement that, but in a test match between Fairy-Max and Pulsar I broadcasted Shatraj proved such a boring game that I never bothered...

Shatranj is very slow, but as I understand it it's less boring if you actually know what you're doing (aiming for strong squares, not count on promotion to Ferz to win you the game instantly, that sort of thing). Sjaak does none of those, but it could...

ChessV plays Shatranj, though. (There is an unreleased version of ChessV that plays Spartan, but its author has disappeared from the radar, so I cannot ask his permission to share it.)

Isn't ChessV's source available under the GPL?

EDIT: even if it is, the original author can release newer versions under a different licence because he holds the copyright. I was just thinking that someone else could pick it up.

[hgm]

I conducted a Xiangqi broadcast in 2010 ('The Xiangqi Clash'), with 33 engines. This was even before I made UCI2WB and could also run the UCI engines (mostly commercials, but very strong). So they were only WB, Qianhong and UCCI engines. See:

http://www.talkchess.com/forum/viewtopic.php?t=35511