CCRL update (28th March 2008)

[Graham Banks]

The March 28th update of the CCRL Rating Lists and Statistics is now available for viewing at:
http://www.computerchess.org.uk/ccrl/4040/

The list gets updated periodically during the week and these updates can be viewed here:
http://www.computerchess.org.uk/ccrl/4040.live/
Please be aware that no game downloads are available from this live link.

The links to the various rating lists can be found just beneath the default Best Versions list.
For example there is a 32-bit Single CPU list.

Our standard testing is at 40 moves in 40 minutes repeating while our current blitz testing is at both 40 moves in 4 minutes repeating and 40 moves in 12 minutes repeating, all adjusted to the AMD64 X2 4600+ (2.4GHz).

Currently active testers in our team are:
Graham Banks, Ray Banks, Shaun Brewer, Kirill Kryukov, Dom Leste, Tom Logan, Andreas Schwartmann, Charles Smith, George Speight, Chris Taylor, Chuck Wilson, Gabor Szots and Martin Thoresen.


40/40 Notes

There are currently 111,719 games in our 40/40 database.

Many engines on our list have few games and in many cases their ratings are likely to fluctuate (markedly for some) until a lot more games are played. Therefore no conclusions should be drawn about their strength yet.
To illustrate this point, when an engine has 200 games played, the error margin is still approximately +-40 ELO, after 500 games +-25 ELO, after 1000 games +-17 ELO and even after 2000 games there is a +-13 ELO error margin!
This of course highlights the importance of looking at other rating lists that are also available in order to draw comparisons and get a more accurate overall picture.


4CPU 64-bit Engines

The interest over the coming weeks will be centred on how much of an improvement Hiarcs 12 is over Hiarcs 11 and whether or not it can lay challenge to Rybka 2.3.2a at the top of the table. Intensive testing has already started and Hiarcs 12 will make its first appearance in next weekend's update.

At present, Zappa Mexico II and Naum 3 are neck and neck for second spot 50+ points behind Rybka.

Deep Shredder 11 lies 40+ ELO further back with a safe edge over Deep Fritz 10.1 and Toga II 1.4 beta5c.

Hiarcs 11.2 and Loop M1-T are next in the pecking order, ahead of Glaurung 2.0.1, Bright 0.3a and Deep Junior 10.

Deep Sjeng 2.7 and Scorpio 2.0 are the other well tested engines in this category.


2CPU Engines

With the emphasis of our multi-cpu testing on 4CPU as opposed to 2CPU, there are gaps in this category and some of the engines also require further games.

Little action to report in this category this week, but with the testing of Hiarcs 12 having started, there will be plenty to follow in the coming weeks.

Rybka 2.3.2a expectedly reigns supreme here also.

Naum 3 currently holds a small, but definite edge over Zappa Mexico for second spot.

Deep Shredder is next, comfortably ahead of Deep Fritz 10, Hiarcs 11.2 and Loop M1-T.
The current rating of Toga II 1.4 beta5c means little as it requires more games.

Bright 0.3a and Glaurung 2.0.1 currently have a small edge over Deep Junior 10, although Bright is another of the engines requiring more games.

Chessmaster 11 and Pharaon 3.5.1 are other engines with a reasonable number of games.


Single CPU Engines

Rybka 2.3.2a has an impressive 100 ELO lead over the the closely grouped Deep Shredder 11, Naum 3, Zappa Mexico II and Fritz 11.
Deep Shredder 11 1CPU is 64-bit as opposed to Shredder 11 which can only be run as a 32-bit engine.

Toga II 3.1.2SE (currently the highest rated of the Togas in this category) has a safe margin over Loop 13.6, Fruit 2.3.1 and Hiarcs 11.2.
Thinker 5.1c Passive has only 31 games and its current rating means little.

Not too far further back, Deep Sjeng 2.7, Spike 1.2 Turin and Glaurung 2.0.1 are very even in strength.
Bright 0.3a could well join this group, but needs many more games before we can be certain.

Junior 10.1 is sandwiched on its own between the group just mentioned and the group of engines below that includes Ktulu 8.0, Chess Tiger 2007.1 and SmarThink 1.00.
Frenzee Feb08 looks likely to establish itself in the latter group, although it still needs a lot more games.

Chessmaster 11, Scorpio 2.0, Booot 4.14.0, Alaric 707 and Movei 00.8.438 (10 10 10) comprise the next group of engines ahead of SlowChess Blitz WV2.1, E.T Chess 13.01.08, Ruffian 2.1.0, Delfi 5.2, WildCat 8, Sloppy 0.2.0, Gandalf 6 and Pro Deo 1.6b.

For Chessmaster enthusiasts, the testing of various settings to see how much gain can be made over the default settings is in full swing and can be followed here:
http://kirr.homeunix.org/chess/discussi ... f=7&t=3054
A Chessmaster tournament will be run soon, pitting the best of the bunch against each other.


Free Single CPU Engines

Toga II 3.1.2SE has possibly overtaken Rybka 1.0 as the top free engine, but it is very close.
However, if Ryan releases his latest Fruit, it is almost certain to take the crown!

As things stand, Fruit 2.3.1 comes in third ahead of Spike 1.2 Turin, Glaurung 2.0.1 and Bright 0.3a.
Thinker 5.1c Passive requires many more games before we can tell exactly where it stands in relation to these engines.

Naum 2.0 and Frenzee Feb08 are 40+ ELO further back.

Scorpio 2.0, Booot 4.14.0, Alaric 707 and Movei 00.8.438 (10 10 10) come in next, ahead of SlowChess Blitz WV2.1, E.T Chess 13.01.08, Zappa 1.1, Delfi 5.2, WildCat 8, Sloppy 0.2.0 and Pro Deo 1.6b.

Special mention should be made regarding Alfil 8.1.1 which seems to be a massive improvement over previous versions. Well done to the author, Enrique Sanchez.
Although we've not started testing GarboChess 2.10 yet, we are also looking forward to seeing how much progress that Gary has made with his engine.

We test a very extensive range of amateur engines (currently ranging down to the 2000 ELO level) through a range of tournaments, all of which can be followed in our public forum.
Our aim is of course to ensure that all engines lower on our lists get 200+ games.


Blitz Notes

An update is currently in progress and should be available soon.

There are currently 257,768 games in our 40/4 database.

The latest ratings can be found at one of the following links:
http://computerchess.org.uk/ccrl/404/
http://computerchess.org.uk/ccrl/404.live/

An enormous amount of work goes into the blitz list and it is well worth a visit.
Hiarcs 12 testing has started here also.

Of special interest to some will be the best free 1CPU engines list which is being constructed through a systematic testing approach as mentioned here:
http://kirr.homeunix.org/chess/discussi ... f=7&t=3271


FRC Notes

There are currently 26,200 games in the FRC 40/4 database.

Ray tests only those engines that can play FRC through the Shredder Classic GUI.
If engine authors have a new and stable version of their engine that will run under this GUI, they should contact Ray if they wish to see it tested.

Ray has started testing Hiarcs 12 and it will be interesting to see if it can grab the FRC crown from Shredder 11.
This should keep FRC fans on tenterhooks for a week or two!

Shredder 11 is currently the top available FRC engine (remembering of course that Rybka 2.3.2 FRC has remained private).
Naum 3 has second spot ahead of Hiarcs 11.1 and Loop 10.32f.

For FRC the best list to look at is the pure list.
http://www.computerchess.org.uk/ccrl/404FRC/


Stats/Presentation Notes

The LOS (likelihood of superiority) stats to the right hand side of each rating list tell you the likelihood in percentage terms of each engine being superior to the engine directly below them.

A list of games played this week per engine can be found in the update thread in the CCRL public forum.

All games are available for download by engine, by month or by ECO code.
ELO ratings are now saved in all game databases for those engines that have 200 games or more.

Clicking on an engine name will give details as to opponents played plus homepage links where applicable.

Custom lists of engines can be selected for comparison.

An openings report page lists the number of games played by ECO codes with draw percentage and White win percentage. Clicking on a column heading will sort the list by that column.

[Tony Thomas]

However, if Ryan releases his latest Fruit, it is almost certain to take the crown!

I hope that he release it as well. May be I should ask him for a copy...

[beachknight]

FWIW, I agree with Tony.

Best,

[Norm Pollock]

Hi Graham,

I noticed something that looks like a possible error in the 40/40 file.

FireFly 2.5.0 was just released in Feb 2008.

However all 42 games of FireFly 2.5.0 in the 40/40 file are dated from 2007.03.11 through 2007.03.15.

I hope this is an isolated incident.

-Norm

[Spock]

Thanks for pointing this out. I've looked at this. The Firefly games were all submitted in March 2008, so you need not be concerned about the integrity of the list. They are genuine Firefly games. This is simply an issue of the date on the tester's newly built PC being 1 year out of date, and therefore the date in his pgn is incorrect. I'll point that out to him

[Norm Pollock]

Thanks for pointing this out. I've looked at this. The Firefly games were all submitted in March 2008, so you need not be concerned about the integrity of the list. They are genuine Firefly games. This is simply an issue of the date on the tester's newly built PC being 1 year out of date, and therefore the date in his pgn is incorrect. I'll point that out to him

I hope the tester also checks other games he submitted to the 40/40 collection.

[Spock]

I hope the tester also checks other games he submitted to the 40/40 collection.

Yes, we will try to find any other games and correct the date, although it's not a big issue

[Norm Pollock]

I hope the tester also checks other games he submitted to the 40/40 collection.

Yes, we will try to find any other games and correct the date, although it's not a big issue

Never said it was a big issue. Certainly 500 years from now no one will care. Actually, I don't care now. I'm just pointing it out because I stumbled on it.

[Graham Banks]

I hope the tester also checks other games he submitted to the 40/40 collection.

Yes, we will try to find any other games and correct the date, although it's not a big issue

Never said it was a big issue. Certainly 500 years from now no one will care. Actually, I don't care now. I'm just pointing it out because I stumbled on it.

We appreciate these things being pointed out though Norm, so thanks.

[Spock]

Never said it was a big issue. Certainly 500 years from now no one will care. Actually, I don't care now. I'm just pointing it out because I stumbled on it.

Yes, and we're very grateful for the scrutiny, if there is any sort of potential problem we need to know about it That was very well spotted !!