CCRL 40/40 and 40/4 lists updated (9th November 2013)

[Graham Banks]

The latest CCRL Rating Lists and Statistics are available for viewing from the following links:
http://computerchess.org.uk/ccrl/4040/ (40/40)
http://www.computerchess.org.uk/ccrl/404/ (40/4)
http://www.computerchess.org.uk/ccrl/404FRC/ (FRC 40/4)

Please note that the three lists are often updated separately to each other. The FRC list is only updated when a new engine or engine version is being/has been tested.

The links to the various rating lists can be found just beneath the default Best Versions list (as in this screenshot). Specific 32-bit rating lists are denoted as such to the right of the default list in each category. The default lists contain the 64-bit engines.



Our 40 moves in 40 minutes repeating and 40 moves in 4 minutes repeating are both adjusted to the AMD64 X2 4600+ (2.4GHz).

Be aware that in the early stages of testing, an engine's rating can often fluctuate a lot.
It is strongly advised to look at the many other rating lists available in order to get a more accurate overall picture of an engine's rating relative to others.

The LOS (likelihood of superiority) stats to the right hand side of each rating list tell you the likelihood in percentage terms of each engine being superior to the engine directly below them.

All games are available for download by engine or ECO code. The total games database in its entirety is always available.
The current ELO ratings are saved in all game databases for those engines that have 200 games or more.

Clicking on an engine name will give details as to opponents played plus homepage links where applicable.

Custom lists of engines can be selected for comparison.

An openings report page lists the number of games played by ECO codes with draw percentage and White win percentage. Clicking on a column heading will sort the list by that column.

[Norm Pollock]

Hi Graham,

I do not think that I have brought this up before. But I believe there is an issue of a small number of opponents for quite a few engines in CCRL 40/40.

I have prepared a list of 25 engines (of the 1377 in total) that lack what I consider a reasonable number of different opponents. Because each played against so few different opponents, CCRL is basing its ratings of these engines on a limited cross-section of competition.

In this list, each engine played over 200 games, yet played against less than 10 different engines of the other 1376 engines besides itself.

Code: Select all

          Name                    #Games   #White   #Black  #Opponents

Arasan 13.2 64-bit                  216      108      108         9
Chessmaster 11 2CPU                 259      131      128         7
Crafty 23.1 64-bit 4CPU             291      147      144         6
Crafty 23.3 64-bit                  248      124      124         8
Critter 0.80 64-bit 4CPU            226      113      113         8
Critter 0.90 64-bit 4CPU            286      143      143         7
Critter 1.01 64-bit 4CPU            229      114      115         9
Deep Fritz 10 2CPU                  227      114      113         9
Deep Fritz 11 4CPU                  335      167      168         9
DoubleCheck 2.3.1 64-bit            286      145      141         9
Glaurung 1.2.1 Beren 64-bit         238      116      122         8
Hiarcs 11 4CPU                      281      140      141         8
Myrddin 0.83k 64-bit                264      132      132         9
Naum 4.1 64-bit 4CPU                376      189      187         9
Predateur 2.0                       224      112      112         7
Predateur 2.2 64-bit                216      108      108         9
Protector 1.3.6 64-bit 4CPU         223      112      111         8
Stockfish 1.3.1 64-bit 2CPU         204      106       98         9
Stockfish 1.5.1 64-bit 4CPU         495      246      249         9
Stockfish 1.6.3 64-bit 2CPU         245      121      124         6
Stockfish 2.1.1 64-bit 4CPU         406      204      202         9
Stockfish 2.2 64-bit                202      101      101         8
Thinker 5.2I Passive 64-bit 4CPU    243      122      121         5
Thinker 5.4a Inert 64-bit           340      169      171         9
Tornado 4.3 64-bit                  224      112      112         7

I also want to mention the following engine although it played slightly less than 200 games:

Code: Select all

Rybka 4 64-bit 2CPU                 193       97       96         3

-Norm

[lkaufman]

Hi Graham,

I do not think that I have brought this up before. But I believe there is an issue of a small number of opponents for quite a few engines in CCRL 40/40.

I have prepared a list of 25 engines (of the 1377 in total) that lack what I consider a reasonable number of different opponents. Because each played against so few different opponents, CCRL is basing its ratings of these engines on a limited cross-section of competition.

In this list, each engine played over 200 games, yet played against less than 10 different engines of the other 1376 engines besides itself.

Code: Select all

          Name                    #Games   #White   #Black  #Opponents

Arasan 13.2 64-bit                  216      108      108         9
Chessmaster 11 2CPU                 259      131      128         7
Crafty 23.1 64-bit 4CPU             291      147      144         6
Crafty 23.3 64-bit                  248      124      124         8
Critter 0.80 64-bit 4CPU            226      113      113         8
Critter 0.90 64-bit 4CPU            286      143      143         7
Critter 1.01 64-bit 4CPU            229      114      115         9
Deep Fritz 10 2CPU                  227      114      113         9
Deep Fritz 11 4CPU                  335      167      168         9
DoubleCheck 2.3.1 64-bit            286      145      141         9
Glaurung 1.2.1 Beren 64-bit         238      116      122         8
Hiarcs 11 4CPU                      281      140      141         8
Myrddin 0.83k 64-bit                264      132      132         9
Naum 4.1 64-bit 4CPU                376      189      187         9
Predateur 2.0                       224      112      112         7
Predateur 2.2 64-bit                216      108      108         9
Protector 1.3.6 64-bit 4CPU         223      112      111         8
Stockfish 1.3.1 64-bit 2CPU         204      106       98         9
Stockfish 1.5.1 64-bit 4CPU         495      246      249         9
Stockfish 1.6.3 64-bit 2CPU         245      121      124         6
Stockfish 2.1.1 64-bit 4CPU         406      204      202         9
Stockfish 2.2 64-bit                202      101      101         8
Thinker 5.2I Passive 64-bit 4CPU    243      122      121         5
Thinker 5.4a Inert 64-bit           340      169      171         9
Tornado 4.3 64-bit                  224      112      112         7

I also want to mention the following engine although it played slightly less than 200 games:

Code: Select all

Rybka 4 64-bit 2CPU                 193       97       96         3

-Norm

I have not seen much evidence that an engine rating is much affected by the choice of opponents, as long as the rating spread is modest. In my opinion even five opponents is quite enough if they are all within a hundred elo or so of the engine being tested.

Larry