CCRL update (7th December 2007)

[Graham Banks]

The December 7th update of the CCRL Rating Lists and Statistics is now available for viewing at:
http://www.computerchess.org.uk/ccrl/4040/

The list gets updated periodically during the week and these updates can be viewed here:
http://www.computerchess.org.uk/ccrl/4040.live/
Please be aware that no game downloads are available from this live link.

The links to the various rating lists can be found just beneath the default Best Versions list.
For example there is a 32-bit Single CPU list.

Our standard testing is at 40 moves in 40 minutes repeating while our current blitz testing is at both 40 moves in 4 minutes repeating and 40 moves in 12 minutes repeating, all adjusted to the AMD64 X2 4600+ (2.4GHz).

Currently active testers in our team are:
Graham Banks, Ray Banks, Shaun Brewer, Kirill Kryukov, Dom Leste, Tom Logan, Andreas Schwartmann, Charles Smith, George Speight, Chris Taylor, Chuck Wilson, Gabor Szots and Martin Thoresen.

A big thanks to all testers as usual for their efforts this week.


40/40 Notes

There currently 90,199 games in our 40/40 database.

Many engines on our list have few games and in many cases their ratings are likely to fluctuate (markedly for some) until a lot more games are played. Therefore no conclusions should be drawn about their strength yet.
To illustrate this point, when an engine has 200 games played, the error margin is still approximately +-40 ELO, after 500 games +-25 ELO, after 1000 games +-17 ELO and even after 2000 games there is a +-13 ELO error margin!
This of course highlights the importance of looking at other rating lists that are also available in order to draw comparisons and get a more accurate overall picture.


4CPU 64-bit Engines

Rybka 2.3.2a is over 50 ELO stronger than Zappa Mexico.

Deep Shredder 11 lies 40 points further back in third spot.

Naum 2.2 comes in fourth, not too far behind Deep Shredder 11, but ahead of the evenly matched pair of Deep Fritz 10.1 and Hiarcs 11.1.
Deep Fritz 10.1 is a 60 ELO improvement over Deep Fritz 10.
Hiarcs 11.1 is ahead of both Hiarcs 11.2 and Hiarcs 11.

The remaining well tested engines in order of rating are Loop M1-T, Glaurung 2.0.1, Deep Junior 10 and Deep Sjeng 2.7.
After 250+ games, Glaurung 2.0.1 is currently 60+ ELO ahead of Glaurung 1.2.1.

Bright 0.2c (private) has made a good start and is currently rated ahead of Deep Sjeng 2.7. However, this is based on a small number of games so far.

We have also started testing Scorpio 2.0.


2CPU Engines

With the emphasis of multi-cpu testing on 4CPU as opposed to 2CPU, there are still gaps in this category and some of the engines also require further games. However, the order of strength is almost identical to that described in the 4CPU notes.


Single CPU Engines

Rybka 2.3.2a reigns supreme, over 100 ELO ahead of Fritz 11, Shredder 11 and Zappa Mexico.

Toga II 1.3.1, Naum 2.2, Hiarcs 11.1, Loop M1-T and Fruit 2.3.1 are within a 25 ELO range of each other.
Fruit 2.3.1 and Fruit 051103 seem to be even in strength.
Toga II 1.3.1 definitely seems to be stronger than Toga II 1.3.4.
Hiarcs 11.1 still has a slight edge over Hiarcs 11.2.

After internal discussions, we have changed our engine notes for Strelka.

A little further back again, Spike 1.2 Turin, Deep Sjeng 2.7 and Junior 10 are very close in strength.
Junior 10 seems to be stronger than Junior 10.1.
Glaurung 2.0.1 currently looks likely to establish itself in this group.

Ktulu 8.0, Bright 0.2b (private) and Chess Tiger 2007.1 are 40-50 ELO further back.
We have received a copy of Bright 0.2c which is purportedly stronger, but we haven't had a chance to test it at 40/40 as yet.

Chessmaster 11 now has 600+ games and is 40+ ELO stronger than CM10th Default.
It is also a little stronger than the best CM10th settings.
How much new settings will improve the strength remains to be seen.

SmarThink 1.00, Alaric 707 and Movei 00.8.438 are the next group of engines that are fairly closely matched in strength.
Delfi 5.2 has slipped behind Delfi 5.1 at present, but that could reverse as more games are played.
We have started testing Movei 00.8.438 (10 10 10) to test Uri's theory that it is stronger than the default settings. It has made a good start, but many more games are required to see if its rating holds.

Scorpio 2.0 has been disappointing to date and lies 40 ELO adrift of the previous version after 163 games, around the same level as Ruffian 2.1.0 and Gandalf 6. This could of course change as more games are played.


Free Single CPU Engines

Each of the top five engines has roughly a 20-25 ELO separation over the engine below them.
Rybka 1.0 64-bit is still the top free engine ahead of Toga II 1.3.1.
Fruit 2.3.1 and Spike 1.2 Turin are third and fourth respectively with Glaurung 2.0.1 looking likely to take over fifth spot ahead of Naum 2.0.

Alaric 707, Movei 0.08.438, SlowChess Blitz WV2.1, Zappa 1.1, Delfi 5.2 and WildCat 7 are within a 20 ELO point range of each other.

We have just started testing Scorpio 2.0 and Pro Deo 1.6. Both have made disappointing starts, but this could change with more games.

Other new engines that are still in the early stages of testing are Crafty 21.6, BugChess2 1.5.2, Sloppy 0.1.1 and Hamsters 0.6.
Twisted Logic 20071202, Typhoon 1.00-346, Xpdnt 071115, Cyrano 0.2d, Little Thought 1.02, NanoSzachy 3.0 and Counter 0.7 are in line to get a run also over the coming months.

We test a very extensive range of amateur engines through our Amateur Championship divisions (32-bit 1CPU) plus other tournaments, all of which can be followed in our public forum.

Our aim is of course to ensure that all engines lower on our lists get 200+ games.


Blitz Notes

There are currently 205,792 games in our 40/4 database.

The 40/4 update is usually done separately to our 40/40 update.
The latest ratings can be found at one of the following links:
http://computerchess.org.uk/ccrl/404/
http://computerchess.org.uk/ccrl/404.live/

An enormous amount of work goes into the blitz list and it is well worth a visit.


FRC Notes

There are currently 23,900 games in the FRC 40/4 database.

Ray tests only those engines that can play FRC through the Shredder Classic GUI.
If engine authors have a new and stable version of their engine that will run under this GUI, they should contact Ray if they wish to see it tested.

Shredder 11 has now taken over second spot behind Rybka 2.3.2 FRC (private), an impressive 80 ELO ahead of Hiarcs 11.1 and Naum 2.2.

Ray has just finished testing Glaurung 2.0.1 and it has made a 60 ELO gain over the previous version, slotting into seventh place on the pure list.

Hamsters 0.6 is likely to be tested soon.

For FRC the best list to look at is the pure list.
http://www.computerchess.org.uk/ccrl/404FRC/


Stats/Presentation Notes

The LOS stats to the right hand side of each rating list are "likelihood of superiority" stats. They tell you the likelihood in percentage terms of each engine being superior to the engine directly below them.

A list of games played this week per engine can be found in the update thread in the CCRL public forum, accessible through the link given at the top of this post.

All games are available for download through the link given at the top of this post. They can be downloaded by engine or by month.
ELO ratings are now saved in all game databases for those engines that have 200 games or more.

Clicking on an engine name will give details as to opponents played plus homepage links where applicable.
Clicking on an engine name now also gives you a ratings history graph for that engine over time (a bit further down the page). The green line is the actual rating. The red lines are the upper and lower error bars, and the blue line represents the number of games.

Custom lists of engines can be selected for comparison.

An openings report page (link at bottom of index page) lists the number of games played by ECO codes with draw percentage and White win percentage. Clicking on a column heading will sort the list by that column.
Games can now be downloaded by ECO code.

[Norm Pollock]

"After internal discussions, we have changed our engine notes for Strelka."

What notes? Any changes? I do not see Stelka on the "Best Engines" list, but I see her on the "Complete List".

[Graham Banks]

"After internal discussions, we have changed our engine notes for Strelka."

What notes? Any changes? I do not see Stelka on the "Best Engines" list, but I see her on the "Complete List".

You need to click on the engine name to get the details.

The legal position of Strelka is untested, and therefore CCRL as a group does not offer any opinion. This is constantly under review. Strelka is playing almost identically to Rybka 1.0 and this is supported by our correlation statistics and by other independent studies. We don't believe that this similarity is accidental and therefore for the purposes of our rating list, we have included it as part of the Rybka family. The implications of that are that it should not appear on any of the pure lists or best versions lists. It will only be visible on the Complete List or any custom lists. A further effect of this is that the volume of future games for Strelka will be low or probably NIL.

[perejaslav]

What about similarity between Loop & Fruit based engines (Toga)? I saw ponder hit = 70%!!! I want to know your opinion about it? AFAIK some years ago List was excluded from WCCC, and now what?

[Tony Thomas]

What about similarity between Loop & Fruit based engines (Toga)? I saw ponder hit = 70%!!! I want to know your opinion about it? AFAIK some years ago List was excluded from WCCC, and now what?

There it goes again.