CCRL 40/4 lists updated (11th August 2012)

[Graham Banks]

The latest CCRL Rating Lists and Statistics are available for viewing from the following links:
http://computerchess.org.uk/ccrl/4040/ (40/40)
http://www.computerchess.org.uk/ccrl/404/ (40/4)
http://www.computerchess.org.uk/ccrl/404FRC/ (FRC 40/4)

Please note that the three lists are updated separately to each other. The 40/40 and 40/4 lists are updated once every two weeks and alternately to each other. The FRC list is updated when a new engine or engine version is being/has been tested.

The links to the various rating lists can be found just beneath the default Best Versions list (as in this screenshot). Specific 32-bit rating lists are denoted as such to the right of the default list in each category. The default lists contain the 64-bit engines.



Our 40 moves in 40 minutes repeating and 40 moves in 4 minutes repeating are both adjusted to the AMD64 X2 4600+ (2.4GHz).

Be aware that in the early stages of testing, an engine's rating can often fluctuate a lot.
It is strongly advised to look at the many other rating lists available in order to get a more accurate overall picture of an engine's rating relative to others.

The LOS (likelihood of superiority) stats to the right hand side of each rating list tell you the likelihood in percentage terms of each engine being superior to the engine directly below them.

All games are available for download by engine, by month or by ECO code. The download databases by month or ECO code are only updated monthly, but the total games database in its entirety is always available.
The current ELO ratings are saved in all game databases for those engines that have 200 games or more.

Clicking on an engine name will give details as to opponents played plus homepage links where applicable.

Custom lists of engines can be selected for comparison.

An openings report page lists the number of games played by ECO codes with draw percentage and White win percentage. Clicking on a column heading will sort the list by that column.

[lkaufman]

Question: What is the actual hardware and actual time limit used for most of the 40/4 games now? What time limit on some brand new Intel computer, say at 3 GHz, would be closest to what you actually use on average? We're trying to find out why our results for Komodo are consistently better at blitz than those reported by both CCRL and CEGT, even with our opening book modified to me more typical of others. Also, is it possible to see whether the ratings of the top few single-core engines would be much different if only pairings among them were rated?

We appreciate all your hard work.

(same question asked of CEGT)

Thanks,
Larry for Komodo

[Graham Banks]

Question: What is the actual hardware and actual time limit used for most of the 40/4 games now? What time limit on some brand new Intel computer, say at 3 GHz, would be closest to what you actually use on average? We're trying to find out why our results for Komodo are consistently better at blitz than those reported by both CCRL and CEGT, even with our opening book modified to me more typical of others. Also, is it possible to see whether the ratings of the top few single-core engines would be much different if only pairings among them were rated?

We appreciate all your hard work.

(same question asked of CEGT)

Thanks,
Larry for Komodo

Hi Larry,

regarding your last question, you could take a look here for the 1CPU engines - see the top twelve crosstable at the bottom. There are some gaps still though.

http://www.computerchess.org.uk/ccrl/40 ... ons_only=1

[lkaufman]

Question: What is the actual hardware and actual time limit used for most of the 40/4 games now? What time limit on some brand new Intel computer, say at 3 GHz, would be closest to what you actually use on average? We're trying to find out why our results for Komodo are consistently better at blitz than those reported by both CCRL and CEGT, even with our opening book modified to me more typical of others. Also, is it possible to see whether the ratings of the top few single-core engines would be much different if only pairings among them were rated?

We appreciate all your hard work.

(same question asked of CEGT)

Thanks,
Larry for Komodo

Hi Larry,

regarding your last question, you could take a look here for the 1CPU engines - see the top twelve crosstable at the bottom. There are some gaps still though.

http://www.computerchess.org.uk/ccrl/40 ... ons_only=1

Thanks. Although the ratings shown are the normal ones and not those derived from the limited crosstable, I can see by inspection that rating only these matches would not narrow the gap. So the fact that we test only against top-level opponents does not appear to explain the disparity in results. One more question, aside from the time limit question: Regarding tablebases, if you test an engine like Komodo or Stockfish that lacks tablebase access, do you still allow the interface to use its own tablebases so that once a tb position is actually reached, the result will always be the correct one?

[Graham Banks]

Question: What is the actual hardware and actual time limit used for most of the 40/4 games now? What time limit on some brand new Intel computer, say at 3 GHz, would be closest to what you actually use on average? We're trying to find out why our results for Komodo are consistently better at blitz than those reported by both CCRL and CEGT, even with our opening book modified to me more typical of others. Also, is it possible to see whether the ratings of the top few single-core engines would be much different if only pairings among them were rated?

We appreciate all your hard work.

(same question asked of CEGT)

Thanks,
Larry for Komodo

Hi Larry,

regarding your last question, you could take a look here for the 1CPU engines - see the top twelve crosstable at the bottom. There are some gaps still though.

http://www.computerchess.org.uk/ccrl/40 ... ons_only=1

Thanks. Although the ratings shown are the normal ones and not those derived from the limited crosstable, I can see by inspection that rating only these matches would not narrow the gap. So the fact that we test only against top-level opponents does not appear to explain the disparity in results. One more question, aside from the time limit question: Regarding tablebases, if you test an engine like Komodo or Stockfish that lacks tablebase access, do you still allow the interface to use its own tablebases so that once a tb position is actually reached, the result will always be the correct one?

Tablebase access is optional for testers. Some choose not to use them whereas others do.
In my case I use the Fritz GUI plus ChessGUI specifically because they have tablebases adjudication as an option.
I'm blowed if I'm going to waste CPU time at longish time controls, allowing a KR v KR ending play out to a 50 move draw.

[lkaufman]

Question: What is the actual hardware and actual time limit used for most of the 40/4 games now? What time limit on some brand new Intel computer, say at 3 GHz, would be closest to what you actually use on average? We're trying to find out why our results for Komodo are consistently better at blitz than those reported by both CCRL and CEGT, even with our opening book modified to me more typical of others. Also, is it possible to see whether the ratings of the top few single-core engines would be much different if only pairings among them were rated?

We appreciate all your hard work.

(same question asked of CEGT)

Thanks,
Larry for Komodo

Hi Larry,

regarding your last question, you could take a look here for the 1CPU engines - see the top twelve crosstable at the bottom. There are some gaps still though.

http://www.computerchess.org.uk/ccrl/40 ... ons_only=1

Thanks. Although the ratings shown are the normal ones and not those derived from the limited crosstable, I can see by inspection that rating only these matches would not narrow the gap. So the fact that we test only against top-level opponents does not appear to explain the disparity in results. One more question, aside from the time limit question: Regarding tablebases, if you test an engine like Komodo or Stockfish that lacks tablebase access, do you still allow the interface to use its own tablebases so that once a tb position is actually reached, the result will always be the correct one?

Tablebase access is optional for testers. Some choose not to use them whereas others do.
In my case I use the Fritz GUI plus ChessGUI specifically because they have tablebases adjudication as an option.
I'm blowed if I'm going to waste CPU time at longish time controls, allowing a KR v KR ending play out to a 50 move draw.

I suppose most testers would use tb adjudication if it is an option for the same reason as you, but you say some do not. I wonder if this is a measurable factor. Theoretically you could rate Komodo and SF separately depending on whether tb adjudication was practiced, but perhaps you don't have this information. Since I suppose most tests are done with a gui that allows tb adjudication, it seems right to do this as the user with the same gui would have tb access for the actual board position for engines like Komodo and SF.

[Graham Banks]

I suppose most testers would use tb adjudication if it is an option for the same reason as you, but you say some do not. I wonder if this is a measurable factor. Theoretically you could rate Komodo and SF separately depending on whether tb adjudication was practiced, but perhaps you don't have this information. Since I suppose most tests are done with a gui that allows tb adjudication, it seems right to do this as the user with the same gui would have tb access for the actual board position for engines like Komodo and SF.

I suspect that it would likely be some doing blitz testing that might choose not to use tablebases or resignations.
The info wouldn't be available separately, but I really do doubt that it would make a measurable difference.

On an unrelated matter, I tend to agree with Albert that contempt=0 would be your best setting.

[lkaufman]

I suppose most testers would use tb adjudication if it is an option for the same reason as you, but you say some do not. I wonder if this is a measurable factor. Theoretically you could rate Komodo and SF separately depending on whether tb adjudication was practiced, but perhaps you don't have this information. Since I suppose most tests are done with a gui that allows tb adjudication, it seems right to do this as the user with the same gui would have tb access for the actual board position for engines like Komodo and SF.

I suspect that it would likely be some doing blitz testing that might choose not to use tablebases or resignations.
The info wouldn't be available separately, but I really do doubt that it would make a measurable difference.

On an unrelated matter, I tend to agree with Albert that contempt=0 would be your best setting.

Is this opinion based on games played in a Fritz gui? We're pretty sure that -7 is better than zero in our own tester, but it is reported that it causes something to go wrong in Fritz gui.

[Modern Times]

All the 40/4 games were played at 40/3 on Intel hardware, which is obviously faster than the AMD64 4600+ reference hardware

[Graham Banks]

On an unrelated matter, I tend to agree with Albert that contempt=0 would be your best setting.

Is this opinion based on games played in a Fritz gui? We're pretty sure that -7 is better than zero in our own tester, but it is reported that it causes something to go wrong in Fritz gui.

No, just an opinion based on many years of testing many engines. Contempt=0 seems to be best in general.