CCRL 40/15, 40/2 and FRC lists updated (22nd August 2020)

[Graham Banks]

The latest CCRL Rating Lists and Statistics are available for viewing from the following links:
http://ccrl.chessdom.com/ccrl/4040 (40/15)
http://ccrl.chessdom.com/ccrl/404 (40/2)
http://ccrl.chessdom.com/ccrl/404FRC (FRC 40/2)

Please note that the three lists are often updated separately to each other. The FRC list is only updated when a new engine or engine version is being/has been tested.

The links to the various rating lists can be found just beneath the default Best Versions list (as in this screenshot). Specific 32-bit rating lists are denoted as such to the right of the default list in each category. The default lists contain the 64-bit engines.



Our 40 moves in 15 minutes repeating and 40 moves in 2 minutes repeating are both adjusted to the Intel i7 4770k (3.5GHz) using the Stockfish 10 64-bit benchmark.

Be aware that in the early stages of testing, an engine's rating can often fluctuate a lot.
It is strongly advised to look at the many other rating lists available in order to get a more accurate overall picture of an engine's rating relative to others.

Thanks to all of our currently active testers (Graham, Ray, Gabor, Sergio and Volodymyr). Thanks also to Chessdom.com for hosting our website.

40/15 testing this week that I'm aware of will include (with live broadcast port for TLCV noted where applicable):

81st Amateur Series Division 4 Tournament (continuing 16053)
81st Amateur Series Division 5 Tournament (continuing 16002)
Attempted Heist Tournament (finishing Wednesday 16001)
New Tournament (starting Wednesday 16001)
Embla 2.0.7 64-bit Gauntlet (continuing 16063)
Arasan 22.1 64-bit Gauntlet (finishing Wednesday 16064)
Combusken 1.3.0 64-bit Gauntlet (continuing 16065)
FoxSEE 5.1.2 64-bit Gauntlet (starting Wednesday 16066)
Amoeba 3.2 64-bit Gauntlet (continuing 16091)
Igel 2.6.0 64-bit Gauntlet (finishing Wednesday 16092)
Igel 2.7.0 NNUE 64-bit Gauntlet (starting Wednesday 16092)
Fruit Reloaded 15A 64-bit Gauntlet (finishing Wednesday 16093)
Orion 0.7 64-bit Gauntlet (starting Wednesday 16093)
Minic 2.46 64-bit Gauntlet (continuing 16094)
RubiChess 1.8 64-bit Gauntlet (being run by Volodymyr)
Ethereal 12.25 64-bit 4CPU Gauntlet (finishing Tuesday 16083)
Gap Filling 64-bit 4CPU Gauntlet (starting Tuesday 16083)
RubiChess 1.8 64-bit 4CPU Gauntlet (starting today 16084)

40/2 testing since the last update report has included (thanks to Gabor and Sergio):

Pulse 1.7.2 64-bit
Mr Bob 0.7.0 64-bit
Halogen 6 64-bit
Skiull 0.5 64-bit
RubiChess 1.8 64-bit
Tamerlane 0.2
ChessMatrix 1.0.6 64-bit

[Vinvin]

Hi Graham,
Thanks for this update.
Maybe it's the time to test the latest SF-NNUE in 40/15, it improved a lot for 2 months.

[lkaufman]

I know that the actual time limit used on specific hardware is adapted to be equivalent to the stated time control on i7-4770k running Stockfish 10 in the opening position, but I couldn't find instructions for exactly how to run the test and what the results would be on that specific hardware. Could you supply that info or a link to where it might be posted? I sometimes run tests on my own computer at the equivalent time control, but so far I've just crudely estimated the proper tc from benchmarks.

[Graham Banks]

I know that the actual time limit used on specific hardware is adapted to be equivalent to the stated time control on i7-4770k running Stockfish 10 in the opening position, but I couldn't find instructions for exactly how to run the test and what the results would be on that specific hardware. Could you supply that info or a link to where it might be posted? I sometimes run tests on my own computer at the equivalent time control, but so far I've just crudely estimated the proper tc from benchmarks.

Should be somewhere in here:
http://kirill-kryukov.com/chess/discuss ... f=7&t=1486

[lkaufman]

I know that the actual time limit used on specific hardware is adapted to be equivalent to the stated time control on i7-4770k running Stockfish 10 in the opening position, but I couldn't find instructions for exactly how to run the test and what the results would be on that specific hardware. Could you supply that info or a link to where it might be posted? I sometimes run tests on my own computer at the equivalent time control, but so far I've just crudely estimated the proper tc from benchmarks.

Should be somewhere in here:
http://kirill-kryukov.com/chess/discuss ... f=7&t=1486

I had found that myself, but I still don't see anything more than a link to get Stockfish 10 and a statement to use the opening position. I suppose you just check NPS, but do you run just one thread, or all, and what is the comparison number for the reference hardware? How did you determine the proper time control to use for your own hardware?

[Modern Times]

You simply run the Stockfish bench command.

An i7-4770k with Stockfish 10 is as follows:

Total time: 2054
Nodes searched: 3939338
Nodes/second: 1917886

The total time taken is our benchmark. To be sure you are running the right version, the nodes searched 3939338 will always be the same.

The time control calculation is a follows:

T minutes for 40 moves where T = Your time / 2054 x 15 (for 40/15)

[lkaufman]

You simply run the Stockfish bench command.

An i7-4770k with Stockfish 10 is as follows:

Total time: 2054
Nodes searched: 3939338
Nodes/second: 1917886

The total time taken is our benchmark. To be sure you are running the right version, the nodes searched 3939338 will always be the same.

The time control calculation is a follows:

T minutes for 40 moves where T = Your time / 2054 x 15 (for 40/15)

Thanks. I'm surprised that a processor dating back to 2013 scores so well on that benchmark. I had overestimated progress in hardware since then.

[Modern Times]

Thanks. I'm surprised that a processor dating back to 2013 scores so well on that benchmark. I had overestimated progress in hardware since then.

It is a Haswell CPU, and yes I think in recent years the focus has been on energy efficiency and more cores.

[lkaufman]

You simply run the Stockfish bench command.

An i7-4770k with Stockfish 10 is as follows:

Total time: 2054
Nodes searched: 3939338
Nodes/second: 1917886

The total time taken is our benchmark. To be sure you are running the right version, the nodes searched 3939338 will always be the same.

The time control calculation is a follows:

T minutes for 40 moves where T = Your time / 2054 x 15 (for 40/15)

I ran tests on Komodo 14 Skill levels 16 thru 24 following your instructions as if I were rating them for the CCRL blitz list, 200 games minimum for levels 16 thru 19, 300 minimum for levels 20 thru 24, pairing each with CCRL engines of fairly similar strength. Here is what I got: skill 16=1531 17=1658 18=1775, 19=1828, 20=1957, 21=2175, 22=2356, 23=2555, 24=2778. I'm a bit surprised that for the higher levels where depth is increasing by one per level (except two ply from 23 to 24) the elo gains are about a full class per level. While there is increasing randomness with lower levels, it's not that much above level 20. A ply shouldn't be a full class. I don't have an explanation at the moment.