CCRL 40/15, 40/2 and FRC lists updated (17th October 2020)

Discussion of computer chess matches and engine tournaments.

Moderators: Harvey Williamson, bob, hgm

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Graham Banks
Posts: 34513
Joined: Sun Feb 26, 2006 9:52 am
Location: Auckland, NZ

CCRL 40/15, 40/2 and FRC lists updated (17th October 2020)

Post by Graham Banks » Sat Oct 17, 2020 11:37 pm

The latest CCRL Rating Lists and Statistics are available for viewing from the following links:
http://ccrl.chessdom.com/ccrl/4040 (40/15)
http://ccrl.chessdom.com/ccrl/404 (40/2)
http://ccrl.chessdom.com/ccrl/404FRC (FRC 40/2)

Please note that the three lists are often updated separately to each other. The FRC list is only updated when a new engine or engine version is being/has been tested.

The links to the various rating lists can be found just beneath the default Best Versions list (as in this screenshot). Specific 32-bit rating lists are denoted as such to the right of the default list in each category. The default lists contain the 64-bit engines.

Image

Our 40 moves in 15 minutes repeating and 40 moves in 2 minutes repeating are both adjusted to the Intel i7 4770k (3.5GHz) using the Stockfish 10 64-bit benchmark.

Be aware that in the early stages of testing, an engine's rating can often fluctuate a lot.
It is strongly advised to look at the many other rating lists available in order to get a more accurate overall picture of an engine's rating relative to others.

The inclusion or exclusion of engines in our lists should not be taken as our group making a statement about their legality or status. Where there is controversy, it is usually mentioned in the engine notes.

Thanks to all of our currently active testers (Graham, Ray, Gabor, Sergio and Volodymyr). Thanks also to Chessdom.com for hosting our website.

40/15 testing this week that I'm aware of will include (with live broadcast port for TLCV noted where applicable):

82nd Amateur Series Division 1 (finishing Thursday 16094)
82nd Amateur Series Division 2 (finishing today 16002)
82nd Amateur Series Division 3 (finishing Tuesday 16053)
82nd Amateur Series Division 4 (starting today 16002)
82nd Amateur Series Division 5 (starting Tuesday 16053)
Deapair Of The Three Kings Tournament (continuing 16001)
Marvin 4.0.0 64-bit Gauntlet (continuing 16063)
Shallow 2020-09-09 64-bit Gauntlet (continuing 16065)
Mr Bob 0.8.0 64-bit Gauntlet (continuing 16066)
Ethereal 12.75 64-bit Gauntlet (continuing 16091)
Igel 2.8.0 NNUE 64-bit Gauntlet (continuing 16092)
Raven 1.10 64-bit Gauntlet (continuing 16093)
Nemorino 6.00 64-bit Gauntlet (starting Thursday 16094)
SlowChess Blitz Classic 2.3 64-bit 4CPU Gauntlet (continuing 16083)
Stockfish 12 64-bit 4CPU Gauntlet (continuing 16084)
Weiss 1.1 64-bit Gauntlet (will be run by Volodymyr)
Halogen 7 64-bit Gauntlet (will be run by Volodymyr)

40/2 testing since the last update report has included (thanks to Gabor and Sergio):

BBC 1.2 64-bit
BBC 1.1 64-bit
Tucano 9.00 64-bit
Supernova 2.0 64-bit
Nemorino 6.00 64-bit
Stash 21.0 64-bit
Sapeli 1.92 64-bit
Stockfish 12 64-bit
Winter 0.9 64-bit
gbanksnz at gmail.com

User avatar
CMCanavessi
Posts: 906
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: CCRL 40/15, 40/2 and FRC lists updated (17th October 2020)

Post by CMCanavessi » Sun Oct 18, 2020 2:40 am

Hey Graham! Thanx for the new entries!
From what I see, there seems to be something weird going on with Tucano 8. It's at 2893 for single CPU, and 2990 for 4CPU. In both cases, it's very very similar in strength to version 7, or in the middle of 7 and 7.05/7.07, which are separated by around 20 elo.

Now you have Tucano 9 at +141 elo from 8.00, which seems excesive, as the author claimed it's about +70; and +163 from 7.00, which seems quite right (see below).

In my own list (which is usually quite consistent with CCRL), I have them as follows:

Tucano 9.00 3069.8 (+54 from Tucano 8.00, a bit short of +70 claimed from author; and +146 from Tucano 7.00, quite similar to CCRL's +163) ==> still low number of games played, will update later
Tucano 8.00 3016.0 (+92 from Tucano 7.00)
Tucano 7.06 2933.5
Tucano 7.00 2923.4

Looking at the CCRL list, I suspect that you may be accidentally using some version of tucano 7.xx binary as if it was version 8. They are astonishinly similar in strength, even tied at 4CPU. Can't find another explanation for the discrepancy.
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

User avatar
Graham Banks
Posts: 34513
Joined: Sun Feb 26, 2006 9:52 am
Location: Auckland, NZ

Re: CCRL 40/15, 40/2 and FRC lists updated (17th October 2020)

Post by Graham Banks » Sun Oct 18, 2020 3:14 am

I can only speak for the 40/15 testing.
There we have the following entries (Tucano 9.00 is still on my waiting list):

Code: Select all

CCRL 40/15 Rating List - Custom engine selection
1183411 games played by 2723 programs, run by 23 testers
Ponder off, General books (up to 12 moves), 3-4-5 piece EGTB
Time control: Equivalent to 40 moves in 15 minutes on an Intel i7-4770k.
Computed on October 17, 2020 with Bayeselo based on 1'183'411 games
Tested by CCRL team, 2005-2020, http://ccrl.chessdom.com/ccrl/4040/

Rank                 Engine                   Elo   +    -   Score  AvOp  Games
1 Tucano 7.07 64-bit 4CPU                 2941  +33  -33  51.5%   -9.7   291
  Tucano 8.00 64-bit                      2886  +18  -18  48.4%  +12.0   959
  Tucano 7.06 64-bit                      2853  +20  -20  49.1%   +0.6   746
  Tucano 7.07 64-bit                      2842  +17  -17  48.6%  +10.5  1132
  Tucano 7.00 64-bit                      2826  +19  -19  52.2%  -19.0   866
  Tucano 7.05 64-bit                      2824  +29  -28  51.4%   -8.0   388
  Tucano 6.00 64-bit                      2724  +15  -16  47.4%  +16.1  1369
  Tucano 5.00 64-bit                      2679  +18  -18  49.7%   +0.8   987
  Tucano 4.00 64-bit                      2666  +25  -25  50.3%   -1.7   546
  Tucano 3.00 64-bit                      2620  +24  -24  48.1%  +12.6   562
  Tucano 2.00 64-bit                      2573  +30  -31  47.4%  +16.5   345
  Tucano 1.04                             2541  +30  -30  47.2%  +17.1   359
gbanksnz at gmail.com

Terje
Posts: 254
Joined: Tue Nov 19, 2019 3:34 am
Location: https://github.com/TerjeKir/weiss
Full name: Terje Kirstihagen

Re: CCRL 40/15, 40/2 and FRC lists updated (17th October 2020)

Post by Terje » Sun Oct 18, 2020 3:37 am

CMCanavessi wrote:
Sun Oct 18, 2020 2:40 am
Hey Graham! Thanx for the new entries!
From what I see, there seems to be something weird going on with Tucano 8. It's at 2893 for single CPU, and 2990 for 4CPU. In both cases, it's very very similar in strength to version 7, or in the middle of 7 and 7.05/7.07, which are separated by around 20 elo.

Now you have Tucano 9 at +141 elo from 8.00, which seems excesive, as the author claimed it's about +70; and +163 from 7.00, which seems quite right (see below).

In my own list (which is usually quite consistent with CCRL), I have them as follows:

Tucano 9.00 3069.8 (+54 from Tucano 8.00, a bit short of +70 claimed from author; and +146 from Tucano 7.00, quite similar to CCRL's +163) ==> still low number of games played, will update later
Tucano 8.00 3016.0 (+92 from Tucano 7.00)
Tucano 7.06 2933.5
Tucano 7.00 2923.4

Looking at the CCRL list, I suspect that you may be accidentally using some version of tucano 7.xx binary as if it was version 8. They are astonishinly similar in strength, even tied at 4CPU. Can't find another explanation for the discrepancy.
What time control are you testing at?

User avatar
CMCanavessi
Posts: 906
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: CCRL 40/15, 40/2 and FRC lists updated (17th October 2020)

Post by CMCanavessi » Sun Oct 18, 2020 12:27 pm

Terje wrote:
Sun Oct 18, 2020 3:37 am
CMCanavessi wrote:
Sun Oct 18, 2020 2:40 am
Hey Graham! Thanx for the new entries!
From what I see, there seems to be something weird going on with Tucano 8. It's at 2893 for single CPU, and 2990 for 4CPU. In both cases, it's very very similar in strength to version 7, or in the middle of 7 and 7.05/7.07, which are separated by around 20 elo.

Now you have Tucano 9 at +141 elo from 8.00, which seems excesive, as the author claimed it's about +70; and +163 from 7.00, which seems quite right (see below).

In my own list (which is usually quite consistent with CCRL), I have them as follows:

Tucano 9.00 3069.8 (+54 from Tucano 8.00, a bit short of +70 claimed from author; and +146 from Tucano 7.00, quite similar to CCRL's +163) ==> still low number of games played, will update later
Tucano 8.00 3016.0 (+92 from Tucano 7.00)
Tucano 7.06 2933.5
Tucano 7.00 2923.4

Looking at the CCRL list, I suspect that you may be accidentally using some version of tucano 7.xx binary as if it was version 8. They are astonishinly similar in strength, even tied at 4CPU. Can't find another explanation for the discrepancy.
What time control are you testing at?
2+1 with ponder on, mostly.
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

Terje
Posts: 254
Joined: Tue Nov 19, 2019 3:34 am
Location: https://github.com/TerjeKir/weiss
Full name: Terje Kirstihagen

Re: CCRL 40/15, 40/2 and FRC lists updated (17th October 2020)

Post by Terje » Sun Oct 18, 2020 12:30 pm

CMCanavessi wrote:
Sun Oct 18, 2020 12:27 pm
Terje wrote:
Sun Oct 18, 2020 3:37 am
CMCanavessi wrote:
Sun Oct 18, 2020 2:40 am
Hey Graham! Thanx for the new entries!
From what I see, there seems to be something weird going on with Tucano 8. It's at 2893 for single CPU, and 2990 for 4CPU. In both cases, it's very very similar in strength to version 7, or in the middle of 7 and 7.05/7.07, which are separated by around 20 elo.

Now you have Tucano 9 at +141 elo from 8.00, which seems excesive, as the author claimed it's about +70; and +163 from 7.00, which seems quite right (see below).

In my own list (which is usually quite consistent with CCRL), I have them as follows:

Tucano 9.00 3069.8 (+54 from Tucano 8.00, a bit short of +70 claimed from author; and +146 from Tucano 7.00, quite similar to CCRL's +163) ==> still low number of games played, will update later
Tucano 8.00 3016.0 (+92 from Tucano 7.00)
Tucano 7.06 2933.5
Tucano 7.00 2923.4

Looking at the CCRL list, I suspect that you may be accidentally using some version of tucano 7.xx binary as if it was version 8. They are astonishinly similar in strength, even tied at 4CPU. Can't find another explanation for the discrepancy.
What time control are you testing at?
2+1 with ponder on, mostly.
Tucano may have had a poor time management for cyclic time controls as in CCRL. I think I remember it using a lot more time than other engines. Maybe this is the reason for the disparity between the results.

sedicla
Posts: 149
Joined: Fri Jan 07, 2011 11:51 pm
Location: USA

Re: CCRL 40/15, 40/2 and FRC lists updated (17th October 2020)

Post by sedicla » Sun Oct 18, 2020 12:53 pm

Yes, can confirm, version 8.00 had an issue with repeating time control, used too much time at initial moves. I made some adjustments.
I guess it could have affected the ratings...

User avatar
CMCanavessi
Posts: 906
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: CCRL 40/15, 40/2 and FRC lists updated (17th October 2020)

Post by CMCanavessi » Sun Oct 18, 2020 1:29 pm

sedicla wrote:
Sun Oct 18, 2020 12:53 pm
Yes, can confirm, version 8.00 had an issue with repeating time control, used too much time at initial moves. I made some adjustments.
I guess it could have affected the ratings...
Interesting! That explains it, then.
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

Jouni
Posts: 2223
Joined: Wed Mar 08, 2006 7:15 pm

Re: CCRL 40/15, 40/2 and FRC lists updated (17th October 2020)

Post by Jouni » Sun Oct 18, 2020 2:20 pm

Nice to have SF12 multicore data finally!
Jouni

lkaufman
Posts: 4339
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: CCRL 40/15, 40/2 and FRC lists updated (17th October 2020)

Post by lkaufman » Sun Oct 18, 2020 6:38 pm

Jouni wrote:
Sun Oct 18, 2020 2:20 pm
Nice to have SF12 multicore data finally!
So the 40/15 list shows the elo gain from SF11 to SF12 (with NNUE) to be 40 elo on 4 threads or 44 elo on 1 thread. These are very small numbers compared to all other estimates, for examples CEGT 40/20 shows 95 elo. Does anyone have any explanation of why the CCRL gains are so small for SF12? I know that using Bayeselo contracts the gains, but even without that difference the gains would be only around 50. Is this a sign of very bad scaling, or perhaps a function of the choice of opponents, or opening books, or ...?
Komodo rules!

Post Reply