CCRL update (2nd May 2008)

[Graham Banks]

The May 2nd update of the CCRL Rating Lists and Statistics is now available for viewing at:
http://www.computerchess.org.uk/ccrl/4040/

The list usually gets updated once or twice during the week and these updates can be viewed here:
http://www.computerchess.org.uk/ccrl/4040.live/
However, no game downloads are available from this live link.

The links to the various rating lists can be found just beneath the default Best Versions list.
For example there is a 32-bit Single CPU list.

Our standard testing is at 40 moves in 40 minutes repeating and our blitz testing is at 40 moves in 4 minutes repeating, both adjusted to the AMD64 X2 4600+ (2.4GHz).
We have abandoned our 40/12 testing and the link will soon be removed. It is already a mammoth task trying to do a decent job with the other lists.

Currently active testers are:
Graham Banks, Ray Banks, Shaun Brewer, Kirill Kryukov, Tom Logan, Charles Smith, George Speight, Chuck Wilson and Gabor Szots.
A few other testers are currently taking a break, but remain on our team.


40/40 Notes

Be aware that in the early stages of testing, an engine's rating can often fluctuate a lot.
It is strongly advised to also look at the many other rating lists available in order to get a more accurate overall picture of an engine's rating relative to others.


4CPU 64-bit Engines

We've now moved on to testing Hiarcs Paderborn, Naum 3.1 and Toga II 1.4.1SE.
At present none of these engines has enough games to say anything with certainty about their rating.

Rybka 2.3.2a remains top dog, 50+ elo ahead of the very evenly matched Zappa Mexico II and Naum 3.
Whether or not Naum 3.1 can bridge the gap further remains to be seen.

Deep Shredder 11 is 30 elo back, but with a slight edge over Hiarcs 12, Deep Fritz 10.1 and Toga II 1.4 beta5c.
It now appears that there is minimal difference whether Hiarcs 12 has Sharpen PV on or off.
Toga II 1.4.1SE is expected to achieve a higher rating than the well tested Toga II 1.4 beta5c.

Loop M1-T, Bright 0.3a and Glaurung 2.0.1 are the next group of engines, holding a clear advantage over Deep Junior 10 and Deep Sjeng 2.7.

Scorpio 2.0 is the only other well tested most recent engine version in this category and is predictably further back.


2CPU Engines

With the emphasis of our multi-cpu testing on 4CPU as opposed to 2CPU, there are gaps in this list and some of the engines also require further games.
We would welcome applications from further testers prepared to focus on 2CPU, so that we could present a more worthwhile rating list in this category.
However, in all honesty, the relative ratings of the engines in this category is pretty much the same as in the 4CPU list.

Toga II 1.4 beta5c was again the main focus of our testing this week in order to get it well established in the list.
The testing of Hiarcs 12 has been held off until we've completed our 4CPU testing of it.


Single CPU Engines

We are currently running gauntlets involving the following engines - Hiarcs 12 Sharpen PV On, Toga II 3.1.3SE, Zappa Mexico II, Thinker 5.1d Passive and Sloppy 0.2.0 amongst others.
Some of those on the gauntlets to do list are Hiarcs 12 Default, Hiarcs Paderborn, Naum 3.1 and Toga II 1.4.1SE.
Over the coming weeks, all of the engines engaged in running gauntlets will accumulate games at a steady rate.

Rybka 2.3.2a has an impressive 100 ELO lead over the closely grouped Deep Shredder 11, Naum 3, Zappa Mexico II and Fritz 11.
Deep Shredder 11 1CPU is 64-bit as opposed to Shredder 11 which can only be run as a 32-bit engine.
It is interesting to note that using Naum as a 64-bit engine seems to have no advantage strength wise over 32-bit.

Toga II 3.1.2SE (currently the highest rated of the Togas tested in this category) holds an edge over Hiarcs 12 Sharpen PV=On.
Hiarcs 12 could well be stronger with its default Sharpen PV Off setting, but many more games will be required to ascertain this and is on the to do list as previously mentioned.

Next come Loop 13.6 and Fruit 2.3.1, with a comfortable gap between them and the closely grouped Deep Sjeng 2.7, Spike 1.2 Turin, Glaurung 2.0.1, Bright 0.3a and Junior 10.
Thinker 5.1d Passive's current rating means little at this early stage.

There is a good distance back to Ktulu 8.0, Chess Tiger 2007.1, SmarThink 1.00 and Frenzee Feb08.

Chessmaster 11, Movei 00.8.438 (10 10 10), Scorpio 2.0, Alaric 707 and Booot 4.14.0 comprise the next group of engines ahead of E.T Chess 13.01.08, SlowChess Blitz WV2.1, Ruffian 2.1.0, WildCat 8 and Delfi 5.2.
We are still in the process of determining which two or three Chessmaster 11 settings that we will test.


Free Single CPU Engines

Toga II 3.1.2SE (the latest version of Toga that we've so far tested extensively at 40/40) is likely the strongest free engine, ahead of Rybka 1.0.
It is expected that Toga II 1.4.1SE will raise the bar further still.
If Ryan releases a later version of Fruit, the pecking order could well change.
The upcoming new Glaurung could also threaten, so there is an interesting battle looming here!

Fruit 2.3.1 comes in third, a good distance ahead of Spike 1.2 Turin, Glaurung 2.0.1 and Bright 0.3a.
Thinker 5.1d Passive is still in the early stages of testing.

Naum 2.0 and Frenzee Feb08 are 40+ ELO further back.

Movei 00.8.438 (10 10 10), Scorpio 2.0, Alaric 707 and Booot 4.14.0 come in next, ahead of E.T Chess 13.01.08, SlowChess Blitz WV2.1, WildCat 8, Zappa 1.1 and Delfi 5.2.

Sloppy 0.2.0, Pro Deo 1.6b, Colossus 2007d, Pharaon 3.5.1 and Ruffian 1.0.5 are the next group.

We test a very extensive range of amateur engines (currently ranging down to the 2000 ELO level) through a range of tournaments, all of which can be followed in our public forum.
Our aim is of course to ensure that all engines lower on our lists get 200+ games.


Blitz Notes

An enormous amount of work goes into the blitz list and it is well worth a visit.

The latest ratings can be found at one of the following links:
http://computerchess.org.uk/ccrl/404/
http://computerchess.org.uk/ccrl/404.live/

Of special interest to some will be the best free 1CPU engines list which is being constructed through a systematic testing approach as mentioned here:
http://kirr.homeunix.org/chess/discussi ... f=7&t=3271


FRC Notes

Ray tests only those engines that can play FRC through the Shredder Classic GUI.
If engine authors have a new and stable version of their engine that will run under this GUI, they should contact Ray if they wish to see it tested.

Hiarcs 12 comes in third amongst the available engines behind Shredder 11 and Naum 3 (remembering of course that the top engine, Rybka 2.3.2z3, has remained private).

For FRC the best list to look at is the pure list.
http://www.computerchess.org.uk/ccrl/404FRC/


Stats/Presentation Notes

The LOS (likelihood of superiority) stats to the right hand side of each rating list tell you the likelihood in percentage terms of each engine being superior to the engine directly below them.

A list of games played this week per engine can be found in the update thread in the CCRL public forum.

All games are available for download by engine, by month or by ECO code.
ELO ratings are now saved in all game databases for those engines that have 200 games or more.

Clicking on an engine name will give details as to opponents played plus homepage links where applicable.

Custom lists of engines can be selected for comparison.

An openings report page lists the number of games played by ECO codes with draw percentage and White win percentage. Clicking on a column heading will sort the list by that column.

[Shaun]

People might be interested the Blitz update, it includes over 1000 games for Naum 3.1 64-bit 4CPU

see here

also there has been a fair amount of Toga testing including Toga II 1.4.2JD which is just behind Toga II 1.4.1SE however only 400+ games each so not conclusive.

Link

Shaun