CCRL rating lists (17th April 2009)

[Graham Banks]

The latest CCRL Rating Lists and Statistics are available for viewing from the following links:
http://computerchess.org.uk/ccrl/4040.live/ (40/40)
http://www.computerchess.org.uk/ccrl/404/ (40/4)
http://www.computerchess.org.uk/ccrl/404FRC/ (FRC 40/4)

Please note that the three lists are updated separately to each other.
Also please note the live link will no longer necessarily give you the most updated lists. The links given in each update report will be the ones to use.

The links to the various rating lists can be found just beneath the default Best Versions list.
For example there is a 32-bit Single CPU list.

Our 40 moves in 40 minutes repeating and 40 moves in 4 minutes repeating are both adjusted to the AMD64 X2 4600+ (2.4GHz).

Currently active testers are:
Graham Banks, Ray Banks (FRC only), Shaun Brewer, Kirill Kryukov, Dom Leste, Tom Logan, Denis Mendoza, Wassim Saeed, Charles Smith, George Speight and Gabor Szots.

Be aware that in the early stages of testing, an engine's rating can often fluctuate a lot.
It is strongly advised to look at the many other rating lists available in order to get a more accurate overall picture of an engine's rating relative to others.


40/40 Notes

There are currently over 176,000 games in our 40/40 database.


4CPU 64-bit Engines

Rybka 3 is clearly stronger than second placed Naum 4, which in turn is clearly stronger than third placed Deep Fritz 11. (Rybka 2.2n2 is the strongest free engine, just ahead of Deep Fritz 11!)
Zappa Mexico II is in fourth spot, comfortably ahead of Thinker 5.4c Inert, Deep Sjeng WC2008 and Deep Shredder 11.
Hiarcs 12, Toga II 1.4.1SE, Glaurung 2.2 and Bright 0.4a come in next and are very evenly matched in strength.
None of the strong Toga or Glaurung derivatives have been tested in this category.

The relative ratings of the 2CPU engines that have been well tested are pretty much the same as their 4CPU counterparts.


Single CPU Engines

A similar situation exists at the top in this category with Rybka 3 holding a huge lead over second placed Naum 4, which in turn has a huge lead over the evenly matched Thinker 5.4c Inert, Fritz 11 and Zappa Mexico II. (The strongest free engine, Rybka 2.2n2, is behind only Rybka 3 and Naum 4 in strength!)
There is a small gap back to Grapefruit 1.0, Deep Sjeng WC2008, Shredder 11, Toga II 1.4.1SE and Cyclone 3.4.
Hiarcs 12.1 and TheMadPrune 1.1.25 come in next, ahead of Onno 0.12.0 (private), Fruit 2.3.1, Loop 13.6, Stockfish 1.2, Glaurung 2.2 and Bright 0.4a. A 20 elo spread covers this last group.
Onno is likely to be sold as a commercial engine sometime in the near future.
Spike 1.2 Turin and Junior 10 have fallen off the pace, but are still comfortably ahead of the group that includes Ktulu 8, Chess Tiger 2007.1, Twisted Logic 20080620, SmarThink 1.00, Frenzee Feb08, Booot 4.15.0, Delfi 5.4 and the top Chessmaster 11 settings. Crafty 23.0 is still in the early stages of testing and could well join this group of engines.


Free Single CPU Engines

Although we're in the early stages of testing Thinker 5.4c Inert, it looks highly unlikely to challenge Rybka 2.2n2 as the top free engine.
Grapefruit 1.0 currently slots into third spot, narrowly ahead of Toga II 1.4.1SE and Cyclone 3.4.
TheMadPrune 1.1.25 bridges the gap between the previous group and the next, which includes Fruit 2.3.1, Stockfish 1.2, Glaurung 2.2 and Bright 0.4a.
Spike 1.2 Turin is further back, with a clear edge over Twisted Logic 20080620, Frenzee Feb08, Booot 4.15.0 and Delfi 5.4. Crafty 23.0 could possibly end up amongst this group after more testing.

CCRL tests a wide range of free engines, ranging right down to the 2000 elo level. The intention is to get well over 200 games for each of these engines.
Tournaments involving these engines can be followed in our public forum.


Blitz Notes

There are over 437,000 games in the 40/4 database and it is well worth a visit.
Shaun, Gabor, Kirill, Denis and Dom put a lot of work into this list, testing engines in a very well organised and systematic manner.


FRC Notes

Ray currently restricts his testing to FRC only.
Thanks to the much appreciated efforts of Matthias Gemuh, ChessGUI is now able to be used to test all FRC engines, both Winboard/UCI plus Shredder/Arena specific.

There are currently over 52,000 games in the database.

Ray has recently finished testing a number of engines - Bright 0.4a, Homer 1.02, Taktix and ArcBishop80.
He only has Tornado 2.2 on his waiting list at present and is hoping that other engine authors might release FRC versions of their engines in future.

Rybka 3 has a big lead over Naum 4, which in turn has a similarly big lead over the evenly matched Deep Sjeng WC2008 and Shredder 11.
Hiarcs 12.1 comes in next, comfortably ahead of Glaurung 2.2.
Fruit 051103 and Loop 10.32f lie further back, ahead of Bright 0.4a and Spike 1.2 Turin.

For FRC the best list to look at is the pure list.
http://www.computerchess.org.uk/ccrl/404FRC/


Stats/Presentation Notes

The LOS (likelihood of superiority) stats to the right hand side of each rating list tell you the likelihood in percentage terms of each engine being superior to the engine directly below them.

All games are available for download by engine, by month or by ECO code.
ELO ratings are now saved in all game databases for those engines that have 200 games or more.

Clicking on an engine name will give details as to opponents played plus homepage links where applicable.

Custom lists of engines can be selected for comparison.

An openings report page lists the number of games played by ECO codes with draw percentage and White win percentage. Clicking on a column heading will sort the list by that column.

A future development (no set date yet) will be the optional inclusion of engine logos, flags and continent pictures beside each engine name in the rating lists.
An example of what this will look can be seen here in Kirill's special free engines project - KCEC site
The engine logo will function as a link to selection of all engines from the same family. The country flag will be a link to selection of all engines from the same country. The continent picture is a link to selection of all engines from the same continent.
This feature will be implemented once Kirill can find the time to do it.

At present our home link gives no information. This will be addressed in the coming weeks.

[pichy]

http://latestchess.com/showNews.php?id=165

http://www.computerchess.org.uk/ccrl/404FRC/

If you take a look at why it is so hard to catch up with Rybka, simply consider that last year when Rybka rating was 3052, now after 8 months Naum has surpased Rybka's previous year rating of 3052 with 3079, but Rybka went up from 3052 to 3182. Rybka 3 has a big lead over Naum 4, which in turn has a similarly big lead over the evenly matched Deep Sjeng WC2008 and Shredder 11.

PS: Why do I consider these ratings ridiculous in comparison to human IPS Chess960 rating? Simply because every year it seems that the computers are raising their ratings by over 100 points, and if it keep up in 5 more years Rybka will reach an incredible rating over 3650. If that is the case, chess will be solved in less than 10 years where all games will end up in a draw.

[Matthias Gemuh]

http://latestchess.com/showNews.php?id=165

http://www.computerchess.org.uk/ccrl/404FRC/

If you take a look at why it is so hard to catch up with Rybka, simply consider that last year when Rybka rating was 3052, now after 8 months Naum has surpased Rybka's previous year rating of 3052 with 3079, but Rybka went up from 3052 to 3182. Rybka 3 has a big lead over Naum 4, which in turn has a similarly big lead over the evenly matched Deep Sjeng WC2008 and Shredder 11.

PS: Why do I consider these ratings ridiculous in comparison to human IPS Chess960 rating? Simply because every year it seems that the computers are raising their ratings by over 100 points, and if it keep up in 5 more years Rybka will reach an incredible rating over 3650. Do you really think that in a match of 8 games in slow time control of game in 90 minutes against the best human player Rybka would reflect this difference?

Hardware and engines are becoming better every year.
Do you expect elo to stand still (as with humans) ?

The efforts of the CCRL and other testers are very much appreciated.
Everyone criticizing any rating list should start by providing a better one and the accompanying games in pgn format.

Matthias.

[pichy]

http://latestchess.com/showNews.php?id=165

http://www.computerchess.org.uk/ccrl/404FRC/

If you take a look at why it is so hard to catch up with Rybka, simply consider that last year when Rybka rating was 3052, now after 8 months Naum has surpased Rybka's previous year rating of 3052 with 3079, but Rybka went up from 3052 to 3182. Rybka 3 has a big lead over Naum 4, which in turn has a similarly big lead over the evenly matched Deep Sjeng WC2008 and Shredder 11.

PS: Why do I consider these ratings ridiculous in comparison to human IPS Chess960 rating? Simply because every year it seems that the computers are raising their ratings by over 100 points, and if it keep up in 5 more years Rybka will reach an incredible rating over 3650. If that is the case, chess will be solved in less than 10 years where all games will end up in a draw.
quote]


Hardware and engines are becoming better every year.
Do you expect elo to stand still (as with humans) ?
Matthias.

And what is your opinion about chess being solved in 10 years, if the computer ratings continue at this rate



Matthias.

[Matthias Gemuh]

And what is your opinion about chess being solved in 10 years, if the computer ratings continue at this rate

My opinion is that even an elo of 5000 will not solve chess.
That is because elo 5000 will still ocasionally blunder and play an elo 3500 move.

Matthias.

[CThinker]

And what is your opinion about chess being solved in 10 years, if the computer ratings continue at this rate

My opinion is that even an elo of 5000 will not solve chess.
That is because elo 5000 will still ocasionally blunder and play an elo 3500 move.

Matthias.

And, I have yet to see an effective solution against blocked positions.

[Matthias Gemuh]

My opinion is that even an elo of 5000 will not solve chess.
That is because elo 5000 will still ocasionally blunder and play an elo 3500 move.

Matthias.

And, I have yet to see an effective solution against blocked positions.

Right. My dream is the day when an elo 3500 move would be a big blunder .
Poor human GMs would feel like abandoning chess.

In theory, one needs an endlessly large elo to solve chess without examing each node and without playing a sub-optimal move.
I personally would believe the result at a time control of 40moves/5mins, if the engine used is 100000 elo strong

Matthias.

[Spock]

Tornado 2.2 x64 FRC testing is now underway.

[AdminX]

A future development (no set date yet) will be the optional inclusion of engine logos, flags and continent pictures beside each engine name in the rating lists.
An example of what this will look can be seen here in Kirill's special free engines project - KCEC site
The engine logo will function as a link to selection of all engines from the same family. The country flag will be a link to selection of all engines from the same country. The continent picture is a link to selection of all engines from the same continent.
This feature will be implemented once Kirill can find the time to do it.

That layout looks really nice, Kirill is doing a find job!