ShashChess 15.0 64-bit Gauntlet for CCRL 40/15

[Graham Banks]

PGN - http://kirill-kryukov.com/chess/discuss ... p?id=47501

[Graham Banks]

Code: Select all

CCRL 40/15 Rating List - Custom engine selection
1195608 games played by 2746 programs, run by 23 testers
Ponder off, General books (up to 12 moves), 3-4-5 piece EGTB
Time control: Equivalent to 40 moves in 15 minutes on an Intel i7-4770k.
Computed on December 5, 2020 with Bayeselo based on 1'195'608 games
Tested by CCRL team, 2005-2020, http://ccrl.chessdom.com/ccrl/4040/

Rank                 Engine                   Elo   +    -   Score  AvOp  Games
1 ShashChess 15.0 64-bit                  3479  +19  -19  75.3% -168.7   951
  Stockfish 12 64-bit                     3474  +17  -16  75.3% -167.3  1249
  SugaR XPro NN 1.0 64-bit                3419  +15  -15  67.4% -111.4  1443
  ShashChess 11.0 64-bit                  3404  +16  -16  67.2% -107.8  1212

[Terje]

I understand that you can't test SF every time it's patched, but replacing it in the list with a fork that includes about 1 month of those patches (20-25ish elo according to SF progression tests, yet the fork only scored +4) seems off to me.

[Uri Blass]

I understand that you can't test SF every time it's patched, but replacing it in the list with a fork that includes about 1 month of those patches (20-25ish elo according to SF progression tests, yet the fork only scored +4) seems off to me.

I see more than +4 for 4 cores but we clearly need more games because I do not know if the bigger difference is because of noise or becauyse ShashChess is relatively better at longer time control(and 4 cores instead of 1 core is probably equivalent to longer time control).

Note that I remember reading that the fork is supposed to be stronger at long time control and I have no idea if it is correct or not correct.

ShashChess 15.0 64-bit 4CPU 3538 +34 −33 76.7% −178.8 46.6% 296
86.0%
Stockfish 12 64-bit 4CPU 3516 +24 −23 74.7% −160.2 49.8% 598

https://ccrl.chessdom.com/ccrl/4040/rat ... t_all.html

[Graham Banks]

This version of ShashChess was released exactly 4 weeks after Stockfish 12.

[Terje]

This version of ShashChess was released exactly 4 weeks after Stockfish 12.

I know, I checked the commit histories and found which commit in SF the Shash 15 release corresponded to. From SF12 release sept 2. to late sept SF gained close to 25 elo according to their own testing.

[Terje]

I understand that you can't test SF every time it's patched, but replacing it in the list with a fork that includes about 1 month of those patches (20-25ish elo according to SF progression tests, yet the fork only scored +4) seems off to me.

I see more than +4 for 4 cores but we clearly need more games because I do not know if the bigger difference is because of noise or becauyse ShashChess is relatively better at longer time control(and 4 cores instead of 1 core is probably equivalent to longer time control).

Note that I remember reading that the fork is supposed to be stronger at long time control and I have no idea if it is correct or not correct.

ShashChess 15.0 64-bit 4CPU 3538 +34 −33 76.7% −178.8 46.6% 296
86.0%
Stockfish 12 64-bit 4CPU 3516 +24 −23 74.7% −160.2 49.8% 598

https://ccrl.chessdom.com/ccrl/4040/rat ... t_all.html

Even if Shash 15.0 was ~20 elo above SF12, SF dev at that point in time was also ~20 elo above SF12 meaning Shash would be ~equal to SF dev. And as you say, with the sample sizes for 4 cores we can't really conclude much.