CCRL 40/40 and 40/4 lists tidied up

[Graham Banks]

Ray spent many hours getting both lists consistent as far as the naming of engines are concerned.

The latest CCRL Rating Lists and Statistics are available for viewing from the following links:
http://computerchess.org.uk/ccrl/4040/ (40/40)
http://www.computerchess.org.uk/ccrl/404/ (40/4)
http://www.computerchess.org.uk/ccrl/404FRC/ (FRC 40/4)

Please note that the three lists are updated separately to each other. The 40/40 and 40/4 lists are updated once every two weeks and alternately to each other. The FRC list is updated when a new engine or engine version is being/has been tested.

[Vinvin]

I developed a small appli (under windows, FreePascal code) to rename all the players in a PGN based on a matching list.
I'll post this evening (in 12 hours).
I hope it will ease your task !

[Modern Times]

Ray spent many hours getting both lists consistent as far as the naming of engines are concerned.

Not yet, the 404 list tidy-up is still in progress.

[Norm Pollock]

Hi Graham,

The following 64 engines do NOT have ANY Elo tags in the 40/40 "pgn" file, although they do have Elo values on the rating lists. No other engines are missing Elo tags. The number of games involved is 6090 and they can be downloaded from

https://dl.dropboxusercontent.com/u/662 ... ing_tag.7z

Alfil 13.1 64-bit 4CPU
Arasan 15.6 64-bit 4CPU
Betsabe II 1.30 64-bit
Bright 0.4a 2CPU
Capivara LK 0.09a02a 64-bit
Crafty 23.5 64-bit 4CPU
Cupcake 1.0
Cyclone 3.4 2CPU
DanaSah 5.06
Delphil 2.9g 64-bit 4CPU
DiscoCheck 4.2 64-bit
EXchess 6.71b 64-bit
EXchess 7.03b 64-bit
EXchess 7.03b 64-bit 4CPU
EveAnn 1.67
EveAnn 1.71
EveAnn 1.71a
FireFly 2.6.0 64-bit
FireFly 2.6.0 64-bit 4CPU
GNU Chess 5.50 64-bit
Glaurung 2.1 32-bit 2CPU
Glaurung 2.2 32-bit 2CPU
Grapefruit 1.0 32-bit 2CPU
Grapefruit 1.0 64-bit 2CPU
GreKo 10.0 64-bit
Gull 2.1 64-bit
Jazz 721 64-bit
Komodo 5.1 MP 64-bit
MadChess 1.2 64-bit
Myrddin 0.86 64-bit
Naum 4.2 64-bit 2CPU
Nebula 2.0 64-bit
Nebula 2.0 64-bit 4CPU
NirvanaChess 1.1 64-bit
Octochess r4984 64-bit
Octochess r4984 64-bit 4CPU
Pawny 1.0 64-bit
Pepito 1.59 64-bit
Predateur 2.2.1 64-bit
Protector 1.5.0 64-bit 4CPU
RedQueen 1.1.3 64-bit 4CPU
Rodent 0.16 64-bit
Rodent 1.0 64-bit
Rodin 6.0
Rybka 3 32-bit 2CPU
Rybka 4 64-bit 2CPU
Scorpio 2.7.5 64-bit 4CPU
Scorpio 2.7.6 64-bit
Sjakk 2.0 64-bit
Spark 0.3 64-bit 2CPU
Sting SF 3 64-bit 4CPU
Stockfish 1.5.1 32-bit 2CPU
Stockfish 1.6.2s 64-bit 2CPU
Texel 1.02 64-bit
TheMadPrune 1.1.25 2CPU
Thinker 5.2k Passive 2CPU
Thinker 5.2m Passive 2CPU
Thinker 5.4a Inert 32-bit 2CPU
Thinker 5.4a Inert 64-bit 2CPU
Toga II 1.4.1SE6 2CPU
Toga II 3.0 4CPU
Tucano 2.00 64-bit
Vajolet 2.03
iCE 1.0 64-bit

-Norm

[Graham Banks]

Hi Norm,

I'll point this out to Ray.

Thanks,
Graham.

[Vinvin]

Oups, I forgot. Here are the files : http://home.scarlet.be/vincentlejeune/r ... me-pgn.zip

There are 4 files inside :

- The .bat with the command inside : Replace-names-all.bat

- The parameters file : paramfileall.txt
there's 4 lines inside :
1) the source input file location (the original PGN)
2) the destination output file location (processed PGN)
3) the substitutions list location input file
4) output file location for the list of engines with less than 20 games (If i remember well)

- listsub.txt (list of text lines to substitute : all lines are tested, if 1st string is found in player name, the string will be substitute by the 2nd line), currently 757 substitutions used to generate the unified-list.

- RemplPGN.exe (the executable ; source is a simple FreePascal code)

[Graham Banks]

Hi Norm,

I'll point this out to Ray.

Thanks,
Graham.

Kirill explained as follows:

Engines with too few games don't have Elo tag in the pgn file. In 40/40 this probably means 200 games. This is our way of saying that those Elos are not reliable and should not be used for anything. For example, automatic opening book building may use ratings. You would not want to use any rating that is based on 15 games. 200 games requirement is arbitrary, but at least it ensures some minimum quality of rating estimates. (Engines with fewer than 200 games also don't appear in the main list).

[PaulieD]

Hi Graham,

When will Stockfish 3 show up in the 40/4 list? I noticed that you have Komodo 5.1 MP there already and it is newer!
Thanks

[Graham Banks]

Hi Graham,

When will Stockfish 3 show up in the 40/4 list? I noticed that you have Komodo 5.1 MP there already and it is newer!
Thanks

Good question.
Stockfish 3 64-bit has less than 200 games at present. Hopefully it will appear as established in the next update.

Graham.

[Norm Pollock]

Kirill explained as follows:

Engines with too few games don't have Elo tag in the pgn file. In 40/40 this probably means 200 games. This is our way of saying that those Elos are not reliable and should not be used for anything. For example, automatic opening book building may use ratings. You would not want to use any rating that is based on 15 games. 200 games requirement is arbitrary, but at least it ensures some minimum quality of rating estimates. (Engines with fewer than 200 games also don't appear in the main list).[/quote]

Graham and Kiril,

With regard to using the CCRL database for creating an opening book, which Kiril mentions, I want to point out two major issues with that project:

1. Many of the engines that are tested are weak, relatively speaking. This is a wonderful attribute for CCRL testing, but it is not so wonderful for creating an opening book. A quality database for an opening book should be based on the moves and results from the strongest players.

2. Many games in the CCRL database have a wide difference in Elo ratings between the players. This skews the results of the games to the stronger engine even if the weaker engine uses the better opening. Players in each game in the database should be relatively equal in strength, perhaps at most a 100 Elo difference. Otherwise the importance of the opening will be obscured.

The following link has 82 CCRL games where there is a 500+ difference in Elo ratings.

https://dl.dropboxusercontent.com/u/66249444/cf1.7z

-Norm