The Clone Tourney - Rybka BETA wins!

[Ovyron]

http://www.talkchess.com/forum/viewtopic.php?t=19695

I finally had the time to finish this tourney, and I'm kind of disappointed that Toga Chekov was not able to win it (But it performed much better than Fruit.)

Athlon 2.1Ghz 32bit
ChessPartner GUI
1'+ 7" with ponder off
64MB hash each engine
6-men online tablebases (Games adjudicated upon reaching 6 piece endings)
No books with manual book learning*
2 Rounds robin each color, 112 games

Participants:




Crosstable:

Code: Select all

The Clone Tourney
2008.02.17 - 2008.03.14
                              Score     Rybk Toga Spik brig Glau Naum Frui Fren
--------------------------------------------------------------------------------
 1: Rybka 1.0 Beta 32-bit   18.5 / 28   XXXX 0011 =1=1 1=01 0010 =1=1 1=1= 1111
 2: Toga II 3.1.2SE Chekov  17.5 / 28   1100 XXXX 0=00 0111 10== 1=1= 1=1= 1111
 3: Spike 1.2 Turin         16.0 / 28   =0=0 1=11 XXXX 1=00 ==11 1011 0011 =01=
 4: bright-0.2c             16.0 / 28   0=10 1000 0=11 XXXX 010= 1110 0111 1=11
 5: Glaurung 2.0.1          15.0 / 28   1101 01== ==00 101= XXXX 1110 010= 0110
 6: Naum 2.0                11.5 / 28   =0=0 0=0= 0100 0001 0001 XXXX 0111 111=
 7: Fruit 2.3.1             10.5 / 28   0=0= 0=0= 1100 1000 101= 1000 XXXX 0=1=
 8: Frenzee feb08            7.0 / 28   0000 0000 =10= 0=00 1001 000= 1=0= XXXX
--------------------------------------------------------------------------------
112 games: +56 =25 -31

Ratings:

Code: Select all

Rank Name                     Elo    +    - games score oppo. draws 
   1 Rybka 1.0 Beta 32-bit   2838   99   93    28   66%  2737   25% 
   2 Toga II 3.1.2SE Chekov  2820   97   92    28   63%  2740   25% 
   3 bright-0.2c             2795   99   96    28   57%  2744   14% 
   4 Spike 1.2 Turin         2793   96   94    28   57%  2744   29% 
   5 Glaurung 2.0.1          2777   97   96    28   54%  2746   21% 
   6 Naum 2.0                2701   94   98    28   41%  2757   18% 
   7 Fruit 2.3.1             2680   93   98    28   38%  2760   25% 
   8 Frenzee feb08           2596   95  107    28   25%  2772   21% 

(* Manual book learning is when I force different moves against the same opponent, if previous moves were unsuccessful. A draw as white is considered unsuccessful. This was done arbitrarily.)

PGN:

http://www.zshare.net/download/89535306d5fb4d/

[Eelco de Groot]

Thanks for running the tournament Ulysses! It looks like Toga 3.1.2SE Checkov did very well under some special tournament rules. Am I correctly assuming that the draws counted for ½ points in the standings?

Rybka 1.0 Beta is still very strong with these very short timecontrols, it reaches an opinion about positions very fast and does not change its mind very much unless for tactical finds. Also with 6 men tablebases some of its shortcomings in elementary endings will have been covered, that is certainly worth a few saved points.

Ulysses I do not really understand how there can be book learning or manual book learning if none of the engines were using any opening books. Could you explain a little more?

Thanks, Eelco

[Ovyron]

Am I correctly assuming that the draws counted for ½ points in the standings?

Yes, the idea behind giving 0.45 points per draw was to avoid ties in the end (An engine that draws twice gets 0.9 while win and lose gives 1.0, so the engines get different scores) ironically, Fruit and Bright tied after the first two rounds so this method isn't infallible. Also, the engines don't know that a draw counts for 0.45, it's unfair for them so I ditched this scoring system and used the standard in the end.

Rybka 1.0 Beta is still very strong with these very short timecontrols

I see. I also hope that the quality of the games is decent, because I can't go slower than this (Even with these fast time controls it took me almost one month to finish it!)

Ulysses I do not really understand how there can be book learning or manual book learning if none of the engines were using any opening books. Could you explain a little more?

I select the engine's moves manually. The first two rounds were played without books.

To avoid possible duplication of games, on the third and fourth round I force engine moves based on past results against specific opponents.

For example, in round 1 Rybka played 1. Nc3 against Toga and lost, so in the third round I force 1. e4 (But it's the only move I force) and this time Rybka won.

On the second round, Toga plays 1. e4 and Rybka plays ...e5 and loses, so on the fourth round I force 1. e4 c5 (I force e4 for Toga because it has learned that it beats Rybka, and force ...c5 for Rybka to avoid e5 that lost) this time Rybka wins.

So, I simulate the kind of learning that some engines have, by avoiding moves that didn't work before, trying to find one that beats the opponent. I also add the element of "opponent modeling" as the "manual" book is only used against the same opponent (In this case, Rybka still plays 1. Nc3 against engines that Rybka knows to beat with it, but these engines will try a different defense this time.)

I like this much better than pseudo random openings that other testers choose, as this system should be able to show the true potential of an engine after certain amount of games (And it works, as some results were reversed in the 3rd and 4th round, even though I choose the new moves arbitrarily.)

Thank you for your interest! My next tourney will include the Thinker engine, and I'm thinking in adding both the Active and Passive personalities to see how they perform under the same tourney (It will have the same participants than this one, excluding Frenzee, that didn't show any output most of the time.)

[Michael Sherwin]

Am I correctly assuming that the draws counted for ½ points in the standings?

Yes, the idea behind giving 0.45 points per draw was to avoid ties in the end (An engine that draws twice gets 0.9 while win and lose gives 1.0, so the engines get different scores) ironically, Fruit and Bright tied after the first two rounds so this method isn't infallible. Also, the engines don't know that a draw counts for 0.45, it's unfair for them so I ditched this scoring system and used the standard in the end.

Rybka 1.0 Beta is still very strong with these very short timecontrols

I see. I also hope that the quality of the games is decent, because I can't go slower than this (Even with these fast time controls it took me almost one month to finish it!)

Ulysses I do not really understand how there can be book learning or manual book learning if none of the engines were using any opening books. Could you explain a little more?

I select the engine's moves manually. The first two rounds were played without books.

To avoid possible duplication of games, on the third and fourth round I force engine moves based on past results against specific opponents.

For example, in round 1 Rybka played 1. Nc3 against Toga and lost, so in the third round I force 1. e4 (But it's the only move I force) and this time Rybka won.

On the second round, Toga plays 1. e4 and Rybka plays ...e5 and loses, so on the fourth round I force 1. e4 c5 (I force e4 for Toga because it has learned that it beats Rybka, and force ...c5 for Rybka to avoid e5 that lost) this time Rybka wins.

So, I simulate the kind of learning that some engines have, by avoiding moves that didn't work before, trying to find one that beats the opponent. I also add the element of "opponent modeling" as the "manual" book is only used against the same opponent (In this case, Rybka still plays 1. Nc3 against engines that Rybka knows to beat with it, but these engines will try a different defense this time.)

I like this much better than pseudo random openings that other testers choose, as this system should be able to show the true potential of an engine after certain amount of games (And it works, as some results were reversed in the 3rd and 4th round, even though I choose the new moves arbitrarily.)

Thank you for your interest! My next tourney will include the Thinker engine, and I'm thinking in adding both the Active and Passive personalities to see how they perform under the same tourney (It will have the same participants than this one, excluding Frenzee, that didn't show any output most of the time.)

Very impressive Ulysses! It is an idea that should not be allowed to disappear. It should be automated in some GUI.

[Ovyron]

Very impressive Ulysses! It is an idea that should not be allowed to disappear. It should be automated in some GUI.

Haha! Thanks Michael, almost makes me want to come back to engine testing, perhaps when I have less than 40 correspondence chess games running!