CCRL 40/15, 40/2 and FRC lists updated (17th October 2020)

[Graham Banks]

The latest CCRL Rating Lists and Statistics are available for viewing from the following links:
http://ccrl.chessdom.com/ccrl/4040 (40/15)
http://ccrl.chessdom.com/ccrl/404 (40/2)
http://ccrl.chessdom.com/ccrl/404FRC (FRC 40/2)

Please note that the three lists are often updated separately to each other. The FRC list is only updated when a new engine or engine version is being/has been tested.

The links to the various rating lists can be found just beneath the default Best Versions list (as in this screenshot). Specific 32-bit rating lists are denoted as such to the right of the default list in each category. The default lists contain the 64-bit engines.



Our 40 moves in 15 minutes repeating and 40 moves in 2 minutes repeating are both adjusted to the Intel i7 4770k (3.5GHz) using the Stockfish 10 64-bit benchmark.

Be aware that in the early stages of testing, an engine's rating can often fluctuate a lot.
It is strongly advised to look at the many other rating lists available in order to get a more accurate overall picture of an engine's rating relative to others.

The inclusion or exclusion of engines in our lists should not be taken as our group making a statement about their legality or status. Where there is controversy, it is usually mentioned in the engine notes.

Thanks to all of our currently active testers (Graham, Ray, Gabor, Sergio and Volodymyr). Thanks also to Chessdom.com for hosting our website.

40/15 testing this week that I'm aware of will include (with live broadcast port for TLCV noted where applicable):

82nd Amateur Series Division 1 (finishing Thursday 16094)
82nd Amateur Series Division 2 (finishing today 16002)
82nd Amateur Series Division 3 (finishing Tuesday 16053)
82nd Amateur Series Division 4 (starting today 16002)
82nd Amateur Series Division 5 (starting Tuesday 16053)
Deapair Of The Three Kings Tournament (continuing 16001)
Marvin 4.0.0 64-bit Gauntlet (continuing 16063)
Shallow 2020-09-09 64-bit Gauntlet (continuing 16065)
Mr Bob 0.8.0 64-bit Gauntlet (continuing 16066)
Ethereal 12.75 64-bit Gauntlet (continuing 16091)
Igel 2.8.0 NNUE 64-bit Gauntlet (continuing 16092)
Raven 1.10 64-bit Gauntlet (continuing 16093)
Nemorino 6.00 64-bit Gauntlet (starting Thursday 16094)
SlowChess Blitz Classic 2.3 64-bit 4CPU Gauntlet (continuing 16083)
Stockfish 12 64-bit 4CPU Gauntlet (continuing 16084)
Weiss 1.1 64-bit Gauntlet (will be run by Volodymyr)
Halogen 7 64-bit Gauntlet (will be run by Volodymyr)

40/2 testing since the last update report has included (thanks to Gabor and Sergio):

BBC 1.2 64-bit
BBC 1.1 64-bit
Tucano 9.00 64-bit
Supernova 2.0 64-bit
Nemorino 6.00 64-bit
Stash 21.0 64-bit
Sapeli 1.92 64-bit
Stockfish 12 64-bit
Winter 0.9 64-bit

[CMCanavessi]

Hey Graham! Thanx for the new entries!
From what I see, there seems to be something weird going on with Tucano 8. It's at 2893 for single CPU, and 2990 for 4CPU. In both cases, it's very very similar in strength to version 7, or in the middle of 7 and 7.05/7.07, which are separated by around 20 elo.

Now you have Tucano 9 at +141 elo from 8.00, which seems excesive, as the author claimed it's about +70; and +163 from 7.00, which seems quite right (see below).

In my own list (which is usually quite consistent with CCRL), I have them as follows:

Tucano 9.00 3069.8 (+54 from Tucano 8.00, a bit short of +70 claimed from author; and +146 from Tucano 7.00, quite similar to CCRL's +163) ==> still low number of games played, will update later
Tucano 8.00 3016.0 (+92 from Tucano 7.00)
Tucano 7.06 2933.5
Tucano 7.00 2923.4

Looking at the CCRL list, I suspect that you may be accidentally using some version of tucano 7.xx binary as if it was version 8. They are astonishinly similar in strength, even tied at 4CPU. Can't find another explanation for the discrepancy.

[Graham Banks]

I can only speak for the 40/15 testing.
There we have the following entries (Tucano 9.00 is still on my waiting list):

Code: Select all

CCRL 40/15 Rating List - Custom engine selection
1183411 games played by 2723 programs, run by 23 testers
Ponder off, General books (up to 12 moves), 3-4-5 piece EGTB
Time control: Equivalent to 40 moves in 15 minutes on an Intel i7-4770k.
Computed on October 17, 2020 with Bayeselo based on 1'183'411 games
Tested by CCRL team, 2005-2020, http://ccrl.chessdom.com/ccrl/4040/

Rank                 Engine                   Elo   +    -   Score  AvOp  Games
1 Tucano 7.07 64-bit 4CPU                 2941  +33  -33  51.5%   -9.7   291
  Tucano 8.00 64-bit                      2886  +18  -18  48.4%  +12.0   959
  Tucano 7.06 64-bit                      2853  +20  -20  49.1%   +0.6   746
  Tucano 7.07 64-bit                      2842  +17  -17  48.6%  +10.5  1132
  Tucano 7.00 64-bit                      2826  +19  -19  52.2%  -19.0   866
  Tucano 7.05 64-bit                      2824  +29  -28  51.4%   -8.0   388
  Tucano 6.00 64-bit                      2724  +15  -16  47.4%  +16.1  1369
  Tucano 5.00 64-bit                      2679  +18  -18  49.7%   +0.8   987
  Tucano 4.00 64-bit                      2666  +25  -25  50.3%   -1.7   546
  Tucano 3.00 64-bit                      2620  +24  -24  48.1%  +12.6   562
  Tucano 2.00 64-bit                      2573  +30  -31  47.4%  +16.5   345
  Tucano 1.04                             2541  +30  -30  47.2%  +17.1   359

[Terje]

Hey Graham! Thanx for the new entries!
From what I see, there seems to be something weird going on with Tucano 8. It's at 2893 for single CPU, and 2990 for 4CPU. In both cases, it's very very similar in strength to version 7, or in the middle of 7 and 7.05/7.07, which are separated by around 20 elo.

Now you have Tucano 9 at +141 elo from 8.00, which seems excesive, as the author claimed it's about +70; and +163 from 7.00, which seems quite right (see below).

In my own list (which is usually quite consistent with CCRL), I have them as follows:

Tucano 9.00 3069.8 (+54 from Tucano 8.00, a bit short of +70 claimed from author; and +146 from Tucano 7.00, quite similar to CCRL's +163) ==> still low number of games played, will update later
Tucano 8.00 3016.0 (+92 from Tucano 7.00)
Tucano 7.06 2933.5
Tucano 7.00 2923.4

Looking at the CCRL list, I suspect that you may be accidentally using some version of tucano 7.xx binary as if it was version 8. They are astonishinly similar in strength, even tied at 4CPU. Can't find another explanation for the discrepancy.

What time control are you testing at?

[CMCanavessi]

Hey Graham! Thanx for the new entries!
From what I see, there seems to be something weird going on with Tucano 8. It's at 2893 for single CPU, and 2990 for 4CPU. In both cases, it's very very similar in strength to version 7, or in the middle of 7 and 7.05/7.07, which are separated by around 20 elo.

Now you have Tucano 9 at +141 elo from 8.00, which seems excesive, as the author claimed it's about +70; and +163 from 7.00, which seems quite right (see below).

In my own list (which is usually quite consistent with CCRL), I have them as follows:

Tucano 9.00 3069.8 (+54 from Tucano 8.00, a bit short of +70 claimed from author; and +146 from Tucano 7.00, quite similar to CCRL's +163) ==> still low number of games played, will update later
Tucano 8.00 3016.0 (+92 from Tucano 7.00)
Tucano 7.06 2933.5
Tucano 7.00 2923.4

Looking at the CCRL list, I suspect that you may be accidentally using some version of tucano 7.xx binary as if it was version 8. They are astonishinly similar in strength, even tied at 4CPU. Can't find another explanation for the discrepancy.

What time control are you testing at?

2+1 with ponder on, mostly.

[Terje]

Hey Graham! Thanx for the new entries!
From what I see, there seems to be something weird going on with Tucano 8. It's at 2893 for single CPU, and 2990 for 4CPU. In both cases, it's very very similar in strength to version 7, or in the middle of 7 and 7.05/7.07, which are separated by around 20 elo.

Now you have Tucano 9 at +141 elo from 8.00, which seems excesive, as the author claimed it's about +70; and +163 from 7.00, which seems quite right (see below).

In my own list (which is usually quite consistent with CCRL), I have them as follows:

Tucano 9.00 3069.8 (+54 from Tucano 8.00, a bit short of +70 claimed from author; and +146 from Tucano 7.00, quite similar to CCRL's +163) ==> still low number of games played, will update later
Tucano 8.00 3016.0 (+92 from Tucano 7.00)
Tucano 7.06 2933.5
Tucano 7.00 2923.4

Looking at the CCRL list, I suspect that you may be accidentally using some version of tucano 7.xx binary as if it was version 8. They are astonishinly similar in strength, even tied at 4CPU. Can't find another explanation for the discrepancy.

What time control are you testing at?

2+1 with ponder on, mostly.

Tucano may have had a poor time management for cyclic time controls as in CCRL. I think I remember it using a lot more time than other engines. Maybe this is the reason for the disparity between the results.

[sedicla]

Yes, can confirm, version 8.00 had an issue with repeating time control, used too much time at initial moves. I made some adjustments.
I guess it could have affected the ratings...

[CMCanavessi]

Yes, can confirm, version 8.00 had an issue with repeating time control, used too much time at initial moves. I made some adjustments.
I guess it could have affected the ratings...

Interesting! That explains it, then.

[Jouni]

Nice to have SF12 multicore data finally!

[lkaufman]

Nice to have SF12 multicore data finally!

So the 40/15 list shows the elo gain from SF11 to SF12 (with NNUE) to be 40 elo on 4 threads or 44 elo on 1 thread. These are very small numbers compared to all other estimates, for examples CEGT 40/20 shows 95 elo. Does anyone have any explanation of why the CCRL gains are so small for SF12? I know that using Bayeselo contracts the gains, but even without that difference the gains would be only around 50. Is this a sign of very bad scaling, or perhaps a function of the choice of opponents, or opening books, or ...?