CCRL 40/40, 40/4 and FRC lists updated (3rd December 2016)

[Graham Banks]

The latest CCRL Rating Lists and Statistics are available for viewing from the following links:
http://computerchess.org.uk/ccrl/4040/ (40/40)
http://www.computerchess.org.uk/ccrl/404/ (40/4)
http://www.computerchess.org.uk/ccrl/404FRC/ (FRC 40/4)

Please note that the three lists are often updated separately to each other. The FRC list is only updated when a new engine or engine version is being/has been tested.

40/40 testing this week that I'm aware of will include (with live broadcast port for TLCV noted where applicable):

The Elite 2016 1CPU Tournament (continuing 16001)
The Second Wave 2016 Tournament (continuing 16002)
The Third Tier 2016 Tournament (continuing 16053)
Carballo 1.7 64-bit Gauntlet (continuing 16064)
Bobcat 8.0 64-bit Gauntlet (continuing 16065)
K2 0.87 Gauntlet (continuing 16066)  
Houdini 5.01 64-bit Gauntlet (continuing)
Fire 5 64-bit Gauntlet (continuing)
Amoeba 2.0 64-bit Gauntlet (continuing
Shallow r694 64-bit Gauntlet (continuing)
Gap-filling Gauntlets (being run by Charles)
Arasan 19.2 64-bit 4CPU Gauntlet (being run by Charles)
Houdini 5.01 64-bit 4CPU Gauntlet (being run by Nathan) 
Enlightenment 4CPU (continuing 16083)
Gap-filling 4CPU (starting Monday 16084)

40/4 testing since the last update has included:

Ethereal 7.78 64-bit
Embla 0.9.5 64-bit
Galjoen 0.33 64-bit
ProDeo 2.2
Fire 5 64-bit 4CPU
Pedone 1.5 64-bit 4CPU
CCRL Authors 40-4 Swiss

[Carlos777]

About the 40/40 list, do you plan to make previous versions of Komodo play against Stockfish 8 and Houdini 5.01? I think it is inaccurate for ELO rating calculations that only the latest versions of programs compete against each other, meaning Komodo 10 and Komodo 10.1 has not faced those 2 top engines or even Shredder 13 or Fire 5.

http://www.computerchess.org.uk/ccrl/40 ... 4-bit_4CPU

http://www.computerchess.org.uk/ccrl/40 ... 4-bit_4CPU

http://www.computerchess.org.uk/ccrl/40 ... 4-bit_4CPU

For example, in Frank Quisinsky's rating list, he makes a new engine to face all the engines he tests, then he compares how the different versions do, keeping in his pure list the best one. Ingo Bauer do the same. I think this kind of testing is sounder.

[Graham Banks]

Komodo 10.1 64-bit 4CPU is currently playing a gauntlet in 16084 against Stockfish 8 64-bit 4CPU, Houdini 5.01 64-bit 4CPU, Andscacs 0.88 64-bit 4CPU, Deep Shredder 13 64-bit 4CPU, Fire 5 64-bit 4CPU and Booot 6.1 64-bit 4CPU.

If you prefer Ingo's or Frank's list, that's fine. You should look at more than one list anyway, as there is no perfect list.

[Modern Times]

Testing of Bobcat 8.0 at chess960 has just started.

[Carlos777]

Komodo 10.1 64-bit 4CPU is currently playing a gauntlet in 16084 against Stockfish 8 64-bit 4CPU, Houdini 5.01 64-bit 4CPU, Andscacs 0.88 64-bit 4CPU, Deep Shredder 13 64-bit 4CPU, Fire 5 64-bit 4CPU and Booot 6.1 64-bit 4CPU.

If you prefer Ingo's or Frank's list, that's fine. You should look at more than one list anyway, as there is no perfect list.

Thats good. I'm looking forward to your results and also how would be the ranking after this.

I follow all the rating lists to compare their results, don't have preference for any in particular. Ingo uses blitz with ponder on and limited number of engines, Frank uses 50 engines and make them play all against all. It is interesting because the top engines play against the weaker engines. Also CEGT and of course CCRL who tests a lot of engines (not only the >2600 ELO group).

Don't take it the wrong way, it is just a concern/suggestion to make the list more accurate.

[Graham Banks]

Don't take it the wrong way, it is just a concern/suggestion to make the list more accurate.

Hi Carlos.

I didn't take any offence.

It's good to look across several rating lists to get a fuller picture, as you say.

[flok]

Thanks for including Embla 0.9.5!
Very interesting to see how it performs.

[Modern Times]

Testing of Bobcat 8.0 at chess960 has just started.

Now completed.

[AndrewGrant]

Is there a reason that Ethereal7.60 is listed on CCRL40/4, and not Ethereal7.78?

Thanks,
Andrew Grant

[Graham Banks]

Is there a reason that Ethereal7.60 is listed on CCRL40/4, and not Ethereal7.78?

Thanks,
Andrew Grant

Code: Select all

CCRL 40/4 Rating List - Custom engine selection
1374058 games played by 1595 programs, run by 20 testers
Ponder off, General books (up to 12 moves), 3-4-5 piece EGTB
Time control: Equivalent to 40 moves in 4 minutes on Athlon 64 X2 4600+ (2.4 GHz), about 1.5 minutes on a modern Intel CPU.
Computed on December 3, 2016 with Bayeselo based on 1'374'058 games
Tested by CCRL team, 2005-2016, http://computerchess.org.uk/ccrl/404/

Rank               Engine                Elo   +    -   Score  AvOp  Games
  Ethereal 7.78 64-bit               2605  +33  -32  61.8%  -83.9   348
1 Ethereal 7.60 64-bit               2450  +18  -18  50.1%   -1.0  1204
  Ethereal 7.33 64-bit               2165  +40  -38  75.9% -209.2   301

Needs extra games to become established.