Stockfish 10 64-bit Gauntlet for CCRL 40/40

[Graham Banks]

PGN - http://kirill-kryukov.com/chess/discuss ... p?id=43395

[Graham Banks]

Code: Select all

CCRL 40/40 Rating List - Custom engine selection
952777 games played by 2318 programs, run by 22 testers
Ponder off, General books (up to 12 moves), 3-4-5 piece EGTB
Time control: Equivalent to 40 moves in 40 minutes on Athlon 64 X2 4600+ (2.4 GHz), about 15 minutes on a modern Intel CPU.
Computed on December 29, 2018 with Bayeselo based on 952'777 games
Tested by CCRL team, 2005-2018, http://computerchess.org.uk/ccrl/4040/

Rank               Engine                Elo   +    -   Score  AvOp  Games
1 Stockfish 10 64-bit                3381  +15  -15  76.7% -183.8  1742
  Stockfish 9 64-bit                 3367  +12  -12  72.2% -155.3  2495
  Stockfish 8 64-bit                 3301  +11  -11  69.9% -131.1  2670
  Stockfish 7 64-bit                 3245  +13  -13  67.9% -112.6  1809
  Stockfish 6 64-bit                 3225  +15  -15  68.7% -117.8  1461
  Stockfish 5 64-bit                 3193  +16  -16  67.5% -107.2  1222
  Stockfish DD 64-bit                3164  +16  -15  63.3%  -79.8  1257
  Stockfish 4 64-bit                 3122  +18  -18  59.7%  -59.4   915
  Stockfish 3 64-bit                 3097  +16  -16  56.5%  -39.4  1186
  Stockfish 2.3.1 64-bit             3078  +17  -17  57.1%  -43.7  1065
  Stockfish 2.2.2 64-bit             3073  +20  -20  54.5%  -31.9   792
  Stockfish 2.2 64-bit               3064  +36  -36  57.2%  -43.9   222
  Stockfish 2.1.1 64-bit             3054  +15  -15  51.2%   -6.9  1258
  Stockfish 2.2.2 32-bit             3051  +14  -14  61.9%  -77.6  1591
  Stockfish 2.3.1 32-bit             3040  +17  -17  57.2%  -44.8  1095
  Stockfish 1.9.1 64-bit             3036  +21  -21  60.6%  -72.0   733
  Stockfish 2.0.1 64-bit             3031  +23  -22  57.7%  -48.6   611
  Stockfish 1.8 64-bit               3028  +22  -22  62.7%  -87.4   703
  Stockfish 2.1.1 32-bit             3028  +16  -16  63.8%  -92.7  1362
  Stockfish 2.0.1 32-bit             3013  +18  -17  66.9% -109.4  1074
  Stockfish 1.7.1 64-bit             3009  +22  -22  61.3%  -74.6   669
  Stockfish 1.9.1 32-bit             3000  +17  -17  67.3% -111.3  1124
  Stockfish 1.8 32-bit               2987  +18  -18  67.5% -110.0  1020
  Stockfish 1.7.1 32-bit             2973  +17  -17  67.1% -112.5  1140
  Stockfish 1.6.3 64-bit             2965  +20  -20  58.6%  -54.9   801
  Stockfish 1.6s 64-bit              2952  +29  -29  52.0%  -14.8   346
  Stockfish 1.6.3 32-bit             2933  +17  -17  61.4%  -72.8  1084
  Stockfish 1.5.1 64-bit             2888  +28  -28  54.7%  -29.6   416
  Stockfish 1.5.1 32-bit             2865  +15  -15  55.4%  -35.5  1486
  Stockfish 1.4 64-bit               2849  +19  -19  53.1%  -17.9   844
  Stockfish 1.4 32-bit               2830  +16  -17  53.9%  -25.6  1120
  Stockfish 1.3.1 32-bit             2793  +25  -25  46.1%  +25.3   512
  Stockfish 1.2 Default              2780  +22  -22  45.5%  +27.8   663
  Stockfish 1.01                     2754  +32  -32  52.5%  -19.7   302

[Chris Formula]

Great work Graham! Any plan testing Leela Chess Zero v0.20.1?

[Graham Banks]

Great work Graham! Any plan testing Leela Chess Zero v0.20.1?

It's possible.

[Chessqueen]

Great work Graham! Any plan testing Leela Chess Zero v0.20.1?

Here is a small match from Youtube
https://www.youtube.com/watch?v=_3n1iwetA2Y

[ernest]

Code: Select all

CCRL 40/40 Rating List - Custom engine selection
952777 games played by 2318 programs, run by 22 testers
Ponder off, General books (up to 12 moves), 3-4-5 piece EGTB
Time control: Equivalent to 40 moves in 40 minutes on Athlon 64 X2 4600+ (2.4 GHz), about 15 minutes on a modern Intel CPU.
Computed on December 29, 2018 with Bayeselo based on 952'777 games
Tested by CCRL team, 2005-2018, http://computerchess.org.uk/ccrl/4040/

Rank               Engine                Elo   +    -   Score  AvOp  Games
1 Stockfish 10 64-bit                3381  +15  -15  76.7% -183.8  1742
  Stockfish 9 64-bit                 3367  +12  -12  72.2% -155.3  2495
  Stockfish 8 64-bit                 3301  +11  -11  69.9% -131.1  2670
  Stockfish 7 64-bit                 3245  +13  -13  67.9% -112.6  1809
  Stockfish 6 64-bit                 3225  +15  -15  68.7% -117.8  1461
  Stockfish 5 64-bit                 3193  +16  -16  67.5% -107.2  1222
  Stockfish DD 64-bit                3164  +16  -15  63.3%  -79.8  1257
  Stockfish 4 64-bit                 3122  +18  -18  59.7%  -59.4   915
  Stockfish 3 64-bit                 3097  +16  -16  56.5%  -39.4  1186
  Stockfish 2.3.1 64-bit             3078  +17  -17  57.1%  -43.7  1065
  Stockfish 2.2.2 64-bit             3073  +20  -20  54.5%  -31.9   792
  Stockfish 2.2 64-bit               3064  +36  -36  57.2%  -43.9   222
  Stockfish 2.1.1 64-bit             3054  +15  -15  51.2%   -6.9  1258
  Stockfish 2.2.2 32-bit             3051  +14  -14  61.9%  -77.6  1591
  Stockfish 2.3.1 32-bit             3040  +17  -17  57.2%  -44.8  1095
  Stockfish 1.9.1 64-bit             3036  +21  -21  60.6%  -72.0   733
  Stockfish 2.0.1 64-bit             3031  +23  -22  57.7%  -48.6   611
  Stockfish 1.8 64-bit               3028  +22  -22  62.7%  -87.4   703
  Stockfish 2.1.1 32-bit             3028  +16  -16  63.8%  -92.7  1362
  Stockfish 2.0.1 32-bit             3013  +18  -17  66.9% -109.4  1074
  Stockfish 1.7.1 64-bit             3009  +22  -22  61.3%  -74.6   669
  Stockfish 1.9.1 32-bit             3000  +17  -17  67.3% -111.3  1124
  Stockfish 1.8 32-bit               2987  +18  -18  67.5% -110.0  1020
  Stockfish 1.7.1 32-bit             2973  +17  -17  67.1% -112.5  1140
  Stockfish 1.6.3 64-bit             2965  +20  -20  58.6%  -54.9   801
  Stockfish 1.6s 64-bit              2952  +29  -29  52.0%  -14.8   346
  Stockfish 1.6.3 32-bit             2933  +17  -17  61.4%  -72.8  1084
  Stockfish 1.5.1 64-bit             2888  +28  -28  54.7%  -29.6   416
  Stockfish 1.5.1 32-bit             2865  +15  -15  55.4%  -35.5  1486
  Stockfish 1.4 64-bit               2849  +19  -19  53.1%  -17.9   844
  Stockfish 1.4 32-bit               2830  +16  -17  53.9%  -25.6  1120
  Stockfish 1.3.1 32-bit             2793  +25  -25  46.1%  +25.3   512
  Stockfish 1.2 Default              2780  +22  -22  45.5%  +27.8   663
  Stockfish 1.01                     2754  +32  -32  52.5%  -19.7   302

Strange...

Only +14 Elo SF10/SF9 ???

[tpoppins]

Given the error margins and limited development time (compared to the v8-v9 cycle) hardly strange.

[ernest]

Given the error margins and limited development time (compared to the v8-v9 cycle) hardly strange.

Thanks for commenting !

But precisely, the error margins make it hard to allow the +14 Elo to reach the announced +50 Elo...
(1% to 2% probability, maybe)

And perhaps CCRL is finding that the +50 Elo is simply not true !

[Jouni]

Here's rating difference SF10 - SF9 from four lists (single core):

FastGM 41/53/47
Pohl 57
CEGT 59/43
CCRL 16/15

[Modern Times]

I got +52 Elo at chess960, but it is so seriously strong there is a lack of opponents for it.