FGRL, 60 sec + 0.6 sec - 2. Dragon by Komodo Chess 1.0, 7. Komodo 14.1 MCTS, 10. SlowChess Blitz Classic 2.4

[fastgm]

FGRL rating list - 60 sec + 0.6 sec

Dragon by Komodo Chess 1.0 (New: 3462)
Komodo 14.1 MCTS (+-0 to Komodo 14 MCTS)
SlowChess Blitz Classic 2.4 (-8 to SlowChess Blitz Classic 2.3)

http://www.fastgm.de

[lkaufman]

FGRL rating list - 60 sec + 0.6 sec

Dragon by Komodo Chess 1.0 (New: 3462)
Komodo 14.1 MCTS (+-0 to Komodo 14 MCTS)
SlowChess Blitz Classic 2.4 (-8 to SlowChess Blitz Classic 2.3)

http://www.fastgm.de

Although the gain for Dragon over K14.1 in this list is 142 elo points, which sounds good, three other testers are confirming gains of around 200 elo in blitz. I note that Dragon scored 93.25% on this list, whereas other tests are run against much stronger engines. Apparently Dragon is relatively better against strong opposition than it is at winning every game from weak opposition, though I don't know why. I look forward to the 16 core rating for Dragon, which I expect to be much closer to SF12 since SF12 only scored 57.5% on that list, so opposition strength shouldn't be a problem. In my opinion pairing opponents more than 300 elo or so apart is not a good way to rate them, although I understand the desire for consistency.

[AndrewGrant]

FGRL rating list - 60 sec + 0.6 sec

Dragon by Komodo Chess 1.0 (New: 3462)
Komodo 14.1 MCTS (+-0 to Komodo 14 MCTS)
SlowChess Blitz Classic 2.4 (-8 to SlowChess Blitz Classic 2.3)

http://www.fastgm.de

Although the gain for Dragon over K14.1 in this list is 142 elo points, which sounds good, three other testers are confirming gains of around 200 elo in blitz. I note that Dragon scored 93.25% on this list, whereas other tests are run against much stronger engines. Apparently Dragon is relatively better against strong opposition than it is at winning every game from weak opposition, though I don't know why. I look forward to the 16 core rating for Dragon, which I expect to be much closer to SF12 since SF12 only scored 57.5% on that list, so opposition strength shouldn't be a problem. In my opinion pairing opponents more than 300 elo or so apart is not a good way to rate them, although I understand the desire for consistency.

Just hold out until FastGM does his higher TC and core count tests. Eventually he will do 16x cores vs only the top 10 or so engines. That will be good data, not swayed by one's ability to crush -1000 elo engines.

[Raphexon]

FGRL rating list - 60 sec + 0.6 sec

Dragon by Komodo Chess 1.0 (New: 3462)
Komodo 14.1 MCTS (+-0 to Komodo 14 MCTS)
SlowChess Blitz Classic 2.4 (-8 to SlowChess Blitz Classic 2.3)

http://www.fastgm.de

Although the gain for Dragon over K14.1 in this list is 142 elo points, which sounds good, three other testers are confirming gains of around 200 elo in blitz. I note that Dragon scored 93.25% on this list, whereas other tests are run against much stronger engines. Apparently Dragon is relatively better against strong opposition than it is at winning every game from weak opposition, though I don't know why. I look forward to the 16 core rating for Dragon, which I expect to be much closer to SF12 since SF12 only scored 57.5% on that list, so opposition strength shouldn't be a problem. In my opinion pairing opponents more than 300 elo or so apart is not a good way to rate them, although I understand the desire for consistency.

Only the 16 core system of FastGM has AVX2.
So the perceived lack of improvement is likely the result of a few missing instruction sets.

[fastgm]

Here are the results of the Top 20 engines.
The gain for Dragon over Komodo 14.1 against stronger engines is 10 elo points more, +152.

Code: Select all

   # Engine                         :    Elo  Error   Games    Points    (%)      W       D      L     D(%)   CFS
 -----------------------------------------------------------------------------------------------------------------
   1 Stockfish 12                   :   3550     10    4750    4075.5   85.80    3456   1239     55   26.08   100
   2 Dragon by Komodo Chess 1.0     :   3482      9    4750    3817.5   80.37    3058   1519    173   31.98   100
   3 ShashChess 12.1.3              :   3412      8    4750    3500.0   73.68    2513   1974    263   41.56   100
   4 Houdini 6                      :   3332      7    4750    3069.0   64.61    2038   2062    650   43.41    63
   5 Komodo 14.1                    :   3330      7    4750    3058.5   64.39    1936   2245    569   47.26   100
   6 Ethereal 12.75                 :   3283      7    4750    2782.0   58.57    1649   2266    835   47.71   100
   7 Komodo 14.1 MCTS               :   3225      6    4750    2418.0   50.91    1266   2304   1180   48.51    95
   8 Nemorino 6.00                  :   3217      7    4750    2367.5   49.84    1264   2207   1279   46.46   100
   9 Fire 7.1                       :   3200      7    4750    2262.0   47.62    1073   2378   1299   50.06    53
  10 Xiphos 0.6                     :   3200      7    4750    2259.5   47.57    1081   2357   1312   49.62    67
  11 SlowChess Blitz Classic 2.4    :   3197      7    4750    2245.5   47.27    1127   2237   1386   47.09   100
  12 Igel 2.8.0 NNUE                :   3169      7    4750    2067.0   43.52    1002   2130   1618   44.84   100
  13 RofChade 2.3                   :   3150      7    4750    1950.5   41.06     842   2217   1691   46.67   100
  14 Schooner 2.2                   :   3115      7    4750    1732.0   36.46     630   2204   1916   46.40    76
  15 Deep Shredder 13               :   3111      7    4750    1708.5   35.97     651   2115   1984   44.53    72
  16 Laser 1.7                      :   3108      7    4750    1690.5   35.59     661   2059   2030   43.35    92
  17 Defenchess 2.2                 :   3101      7    4750    1648.5   34.71     627   2043   2080   43.01    79
  18 RubiChess 1.8                  :   3097      7    4750    1624.0   34.19     630   1988   2132   41.85    51
  19 Booot 6.4                      :   3096      7    4750    1623.5   34.18     611   2025   2114   42.63    77
  20 Fritz 17                       :   3093      7    4750    1600.5   33.69     613   1975   2162   41.58   ---

White advantage = 35.74 +/- 1.19
Draw rate (equal opponents) = 55.44 % +/- 0.27

Games        : 47.500 (finished)

White Wins   : 15.313 (32.2 %)
Black Wins   : 11.415 (24.0 %)
Draws        : 20.772 (43.7 %)

White Score  : 54.1 %
Black Score  : 45.9 %

[lkaufman]

FGRL rating list - 60 sec + 0.6 sec

Dragon by Komodo Chess 1.0 (New: 3462)
Komodo 14.1 MCTS (+-0 to Komodo 14 MCTS)
SlowChess Blitz Classic 2.4 (-8 to SlowChess Blitz Classic 2.3)

http://www.fastgm.de

Although the gain for Dragon over K14.1 in this list is 142 elo points, which sounds good, three other testers are confirming gains of around 200 elo in blitz. I note that Dragon scored 93.25% on this list, whereas other tests are run against much stronger engines. Apparently Dragon is relatively better against strong opposition than it is at winning every game from weak opposition, though I don't know why. I look forward to the 16 core rating for Dragon, which I expect to be much closer to SF12 since SF12 only scored 57.5% on that list, so opposition strength shouldn't be a problem. In my opinion pairing opponents more than 300 elo or so apart is not a good way to rate them, although I understand the desire for consistency.

Only the 16 core system of FastGM has AVX2.
So the perceived lack of improvement is likely the result of a few missing instruction sets.

I didn't know that these games were run on a system without AVX2. That is a very big deal in Dragon, so much of the difference between the +152 elo reported when considering just the top 20 engines and the roughly +200 reported by three other testers is due to AVX2, while the rest is likely due to a much narrower range of elo ratings in the opposing engines.

[Jouni]

Is this AVX2 thing only Ryzen? In my Intel tests bmi has been always the fastest version of SF NNUE!

[Raphexon]

Is this AVX2 thing only Ryzen? In my Intel tests bmi has been always the fastest version of SF NNUE!

Every CPU that supports BMI2 supports AVX2. (except for maybe a few weird chips)
BMI2 compiles are compiled with the assumption that the CPU will also be able to utilize AVX2 in mind. So if you're using a BMI2 compile, you're also using the AVX2 instruction set.

[jonkr]

The result with Slow 2.4 slightly below 2.3 and 2.2 does come as a surprise even knowing the gains might not show up so much against weaker opponents. In SPCC and in the CEGT forum I see a 20+ elo gain for 2.4 vs 2.3, and versus the older 2.2 I got +42 elo standard, +51 FRC in my own testing. Not having AVX2 should only matter slightly since it's just endgame nets will be about 30% slower, which even without AVX2 was still stronger than no nets. Will see how it does in the other tests. (Just in case : Slow64-noAvx.exe should be the one to use, the Slow64-noPop.exe is like 6-8% slower overall for me, and the Slow64.exe I assume would crash. Also with 2.4 it expects the data files in the same dir as the exe now but the zip has been arranged this way so still just a straight extract.)

[lkaufman]

Is this AVX2 thing only Ryzen? In my Intel tests bmi has been always the fastest version of SF NNUE!

We weren't able to prove that making a bmi version gave any extra speed beyond the avx2 version for Dragon, regardless of the hardware, so we didn't make a special bmi version. Of course that could change in the future.