Stockfish vs. Komodo (fast draws) ... a big different!!

[Frank Quisinsky]

Hi there,

I am looking in statistics, games ended with move 80 (without resign mode).
Since a while the contempt parameter missed in Stockfish.

Should be clear for all of us that Stockfish is the number 1 and Dragon by Komodo the number 2 in the big group of CPU-Engines.

I wrote this for some weeks ...
Stockfish cloner produced some Elo more with an intelligent contempt parameter if we compare with the original. I think this room for "cloners" the Stockfish team must not give. Stockfish can be much stronger with an intelligent contempt!!

Here the statisticis after 6590 games, after round 8 of 50 from the still running FCP Tourney-2022:

Which engine give to fast draw ...

- with WHITE pieces -

Stockfish on Position 1 in TOP-41
Komodo on Position 41 in TOP-41

Really very crazy ...

Important is to looking on move average:

The different!


E. FAST WHITE DRAWS BY MOVE AVERAGE:
Filter: 20 to 80 moves, Games that have all type of results

1. Stockfish 151121 NN x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 49 5 92 5.4%

2. rofChade 2.310 NN x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 51 30 66 45.5%

3. Marvin 5.2.0 NN AVX2 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 52 46 80 57.5%

4. Orion 0.8 NN FMA x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 53 47 87 54.0%

5. DanaSah 9.0 NN AVX2 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 53 49 84 58.3%

6. Ethereal 13.25 NN PEXT x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 53 21 62 33.9%

7. Igel 3.0.10 NN BMI2 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 53 55 90 61.1%

8. Booot 6.5 POPCNT x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 54 38 71 53.5%

9. Seer 2.4.0 NN AVX2 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 54 34 53 64.2%

10. Clover 2.4 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 54 48 72 66.7%

11. Zahak 8.6 AMD x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 54 35 74 47.3%

12. Wasp 5.00 NN AVX2 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 55 54 82 65.9%

13. Halogen 10 NN PEXT x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 56 58 85 68.2%

14. Berserk 7 NN PEXT x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 56 18 47 38.3%

15. Winter 0.9 BMI2 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 56 45 75 60.0%

16. Stash 31.16 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 56 35 61 57.4%

17. Dark Toga 1.1 NN AVX2 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 56 18 49 36.7%

18. RubiChess 2.2 NN x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 56 43 87 49.4%

19. Andscacs 0.95.123 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 57 39 61 63.9%

20. SlowChess Blitz 2.8 NN AVX2 x
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 57 25 85 29.4%

21. Revenge 1.0 NN x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 57 40 86 46.5%

22. Lc0 0.28.0 NN CPU-dnnl
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 57 45 80 56.2%

23. Fire 8 NN MC.3 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 58 30 87 34.5%

24. Weiss 2.0 PEXT x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 58 47 80 58.8%

25. GullChess 3.0 Sy BMI2 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 58 31 71 43.7%

26. Fritz 18 (Ginkgo) x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 59 25 60 41.7%

27. Combusken 1.4.0 AMD x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 59 32 68 47.1%

28. Schooner 2.2 XB SSE x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 59 50 79 63.3%

29. Fizbo 2.0 BMI2 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 59 45 78 57.7%

30. Minic 3.17 NN x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 59 38 68 55.9%

31. Chiron 5 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 59 45 70 64.3%

32. Nemorino 6.09 NN x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 59 36 68 52.9%

33. Laser 1.7 BMI2 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 59 47 72 65.3%

34. Defenchess 2.3 dev BMI2 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 60 35 60 58.3%

35. Xiphos 0.6 BMI2 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 60 42 73 57.5%

36. Shredder 13 POPCNT x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 61 45 79 57.0%

37. Demolito 2021-07-09 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 61 25 53 47.2%

38. Koivisto 7.5 NN AVX2 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 61 17 50 34.0%

39. Nirvanachess 2.5 POPCNT x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 61 36 71 50.7%

40. Arasan 23.0.1 NN AVX2 x64
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 62 40 84 47.6%

41. Dragon 2.5.1 by Komodo NN AVX
CATEGORIES MOVE_AVE GAMES TOTAL GAME_RATE
white draw 64 14 100 14.0%

[Frank Quisinsky]

Now, a big deeper ...

Komodo must have a clearly bigger move average Stockfish have (all games, without resign mode).

Let us have a look in the current rating list after 320 games ...

Code: Select all

  FCP Tourney-2022
  ----------------

  Date           : November 27th, 2021 (10:15)
  Games          : 06.560
  Version        : 08.0, after R08
  Elo            : Shredder 13 POPCNT x64 = 3125
  Conditions     : http://www.amateurschach.de/main/_fcp-tourney-2022.htm

  Generated with : Ordo 1.2.6 x64 (Ratings for chess and other games)
  https://sites.google.com/site/gaviotachessengine/ordo
  by Miguel A. Ballicora (Argentina)



   # Player                                 :      Elo  Games  Score%   won  draw  lost  Points  Draw%   Error   OppAvg   OppE  MoveAvg
  -------------------------------------------------------------------------------------------------------------------------------------
  01. Stockfish 151121 NN x64               :  3491.92    320    83.3   213   107     0   266.5   33.4   34.37  3178.05  24.69     82.5
  02. Dragon 2.5.1 by Komodo NN AVX2 x64    :  3470.04    320    81.6   203   116     1   261.0   36.3   31.25  3178.60  24.77     86.1
  03. Fire 8 NN MC.3 x64                    :  3395.31    320    74.8   164   151     5   239.5   47.2   27.88  3180.46  24.85     83.5
  04. SlowChess Blitz 2.8 NN AVX2 x64       :  3372.11    320    72.5   152   160     8   232.0   50.0   26.44  3181.04  24.89     87.7
  05. Koivisto 7.5 NN AVX2 x64              :  3366.12    320    71.9   150   160    10   230.0   50.0   26.85  3181.19  24.88     96.9
  06. Ethereal 13.25 NN PEXT x64            :  3341.37    320    69.2   145   153    22   221.5   47.8   26.75  3181.81  24.88     92.9
  07. Berserk 7 NN PEXT x64                 :  3324.54    320    67.3   128   175    17   215.5   54.7   25.91  3182.23  24.90     94.8
  08. RubiChess 2.2 NN x64                  :  3320.40    320    66.9   127   174    19   214.0   54.4   25.42  3182.34  24.91     86.6
  09. Revenge 1.0 NN x64                    :  3298.76    320    64.4   112   188    20   206.0   58.8   24.91  3182.88  24.93     81.3
  10. Igel 3.0.10 NN BMI2 x64               :  3286.87    320    63.0   110   183    27   201.5   57.2   25.18  3183.18  24.92     81.9
  -------------------------------------------------------------------------------------------------------------------------------------
  11. Seer 2.4.0 NN AVX2 x64                :  3272.56    320    61.3   109   174    37   196.0   54.4   24.67  3183.53  24.93     88.1
  12. rofChade 2.310 NN x64                 :  3266.13    320    60.5   104   179    37   193.5   55.9   24.03  3183.69  24.95     89.4
  13. Nemorino 6.09 NN x64                  :  3264.84    320    60.3   110   166    44   193.0   51.9   25.14  3183.73  24.92     84.8
  14. Arasan 23.0.1 NN AVX2 x64             :  3229.54    320    55.9    86   186    48   179.0   58.1   22.50  3184.61  24.99     86.3
  15. Lc0 0.28.0 NN CPU-dnnl                :  3224.57    320    55.3    84   186    50   177.0   58.1   24.18  3184.73  24.95     88.2
  16. Booot 6.5 POPCNT x64                  :  3198.66    320    52.0    75   183    62   166.5   57.2   23.55  3185.38  24.96     89.5
  17. Xiphos 0.6 BMI2 x64                   :  3172.94    320    48.8    61   190    69   156.0   59.4   23.68  3186.02  24.96     86.6
  18. Weiss 2.0 PEXT x64                    :  3152.12    320    46.1    55   185    80   147.5   57.8   23.38  3186.54  24.97     87.9
  19. Minic 3.17 NN x64                     :  3150.89    320    45.9    56   182    82   147.0   56.9   23.42  3186.58  24.96     89.2
  20. Schooner 2.2 XB SSE x64               :  3148.44    320    45.6    54   184    82   146.0   57.5   23.24  3186.64  24.97     85.5
  -------------------------------------------------------------------------------------------------------------------------------------
  21. Wasp 5.00 NN AVX2 x64                 :  3147.21    320    45.5    45   201    74   145.5   62.8   22.92  3186.67  24.98     86.5
  22. Zahak 8.6 AMD x64                     :  3145.98    320    45.3    68   154    98   145.0   48.1   23.61  3186.70  24.96     85.1
  23. Defenchess 2.3 dev BMI2 x64           :  3129.95    320    43.3    48   181    91   138.5   56.6   22.92  3187.10  24.98     91.0
  24. Shredder 13 POPCNT x64                :  3125.00    320    42.7    48   177    95   136.5   55.3   24.50  3187.22  24.94     84.6
  25. DanaSah 9.0 NN AVX2 x64               :  3123.76    320    42.5    46   180    94   136.0   56.3   24.10  3187.25  24.95     85.0
  26. Fritz 18 (Ginkgo) x64                 :  3120.03    320    42.0    51   167   102   134.5   52.2   23.72  3187.35  24.96     93.0
  27. Clover 2.4 x64                        :  3118.79    320    41.9    39   190    91   134.0   59.4   23.58  3187.38  24.96     87.8
  28. Laser 1.7 BMI2 x64                    :  3117.54    320    41.7    47   173   100   133.5   54.1   23.83  3187.41  24.95     89.2
  29. Marvin 5.2.0 NN AVX2 x64              :  3106.29    320    40.3    41   176   103   129.0   55.0   23.27  3187.69  24.97     86.4
  29. Chiron 5 x64                          :  3106.29    320    40.3    41   176   103   129.0   55.0   23.32  3187.69  24.97     89.4
  -------------------------------------------------------------------------------------------------------------------------------------
  31. Andscacs 0.95.123 x64                 :  3099.99    320    39.5    42   169   109   126.5   52.8   24.04  3187.85  24.95     90.3
  32. Fizbo 2.0 BMI2 x64                    :  3088.58    320    38.1    41   162   117   122.0   50.6   24.06  3188.13  24.95     85.8
  33. Winter 0.9 BMI2 x64                   :  3079.61    320    37.0    40   157   123   118.5   49.1   24.50  3188.36  24.94     86.1
  34. Dark Toga 1.1 NN AVX2 x64             :  3077.04    320    36.7    43   149   128   117.5   46.6   23.70  3188.42  24.96     97.6
  35. Halogen 10 NN PEXT x64                :  3066.65    320    35.5    29   169   122   113.5   52.8   25.02  3188.68  24.92     83.2
  36. Orion 0.8 NN FMA x64                  :  3053.48    320    33.9    27   163   130   108.5   50.9   25.98  3189.01  24.90     79.5
  37. GullChess 3.0 Sy BMI2 x64             :  3049.48    320    33.4    31   152   137   107.0   47.5   24.56  3189.11  24.94     87.8
  38. Stash 31.16 x64                       :  3048.15    320    33.3    29   155   136   106.5   48.4   24.46  3189.14  24.94     91.9
  39. Nirvanachess 2.5 POPCNT x64           :  3035.99    320    31.9    22   160   138   102.0   50.0   25.02  3189.45  24.92     86.7
  40. Combusken 1.4.0 AMD x64               :  3022.20    320    30.3    32   130   158    97.0   40.6   25.84  3189.79  24.90     85.9
  -------------------------------------------------------------------------------------------------------------------------------------
  41. Demolito 2021-07-09 x64               :  3003.77    320    28.3    20   141   159    90.5   44.1   26.30  3190.25  24.89     96.0
  -------------------------------------------------------------------------------------------------------------------------------------

  MoveAvg (average)           : 177.64 / 88.820
  White advantage             : 67.07  +/- 2.85
  Draw rate (equal opponents) : 69.64% +/- 0.81

  Games        :   6.560 (finished)
  White Wins   :   2.076 (31.6%)
  Black Wins   :   1.052 (16.0%)
  Draws        :   3.432 (52.3%)
  Unfinished   :       0

  White Perf.  : 57.8%
  Black Perf.  : 42.2%

  ECO A00-A99  :    1.249 Games (19.0%)
  ECO B00-B99  :    1.486 Games (22.7%)
  ECO C00-C99  :    1.501 Games (22.9%)
  ECO D00-D99  :    1.085 Games (16.5%)
  ECO E00-E99  :    1.239 Games (18.9%)
  

Move average from Komodo is 3.6 moves higher only.
During the other gaming phases Komodo can play more interesting chess.
But to explain this ... I have to give 11 different statistics about it.

Without to do that ...

Shortly:
After exchange pieces in mid-games the pawn structures from Komodo for the endgame is better. Komodo have here a ELO advantage to Stockfish from around 35 points (statistics I made from the versions I am using for the last FCP Tourney-2021). After all I can see Komodo can hold the advantage (have not enough games to the still running FCP Tourney-2022).

Back:
A big deeper ...
Topic is the move average!

Code: Select all

12. Dragon 2.5.1 by Komodo NN AVX  (123+,   42=,   1-)  86.7%

          CATEGORIES   MOVE_AVE    GAMES    TOTAL  GAME_RATE
           all games         64      166
           white won         67       85
           black won         67       38
          white draw         64       14
          black draw         54       28
          white loss         70        1
          black loss          0        0
            all wins         67      123      166      74.1%
           all draws         58       42      166      25.3%
           all loses         70        1      166       0.6%

Code: Select all

35. Stockfish 151121 NN x64        (139+,   36=,   0-)  89.7%

          CATEGORIES   MOVE_AVE    GAMES    TOTAL  GAME_RATE
           all games         65      175
           white won         65       87
           black won         69       52
          white draw         49        5
          black draw         59       31
          white loss          0        0
          black loss          0        0
            all wins         67      139      175      79.4%
           all draws         57       36      175      20.6%
           all loses          0        0      175       0.0%

With white pieces ...
Stockfish have problem with to fast draws games!!

Best
Frank

With the final result ... in my humble opinion ...
For opening analyzes ... to use Dragon is better as to use Stockfish!!

[Frank Quisinsky]

Hi there,

end of the tourney it make more sense to looking in detail.
Should be clear!

But the "to-fast-give-draw" problem with _white-pieces_ from Stockfish ...
With a small numbers of games every time very easy to see with or without statistics.

Best
Frank

[peter]

Stockfish cloner produced some Elo more with an intelligent contempt parameter if we compare with the original. I think this room for "cloners" the Stockfish team must not give. Stockfish can be much stronger with an intelligent contempt!!

Don't know for sure, what you mean, Frank. Contempt in SF dev since quite a while is auto- adapting and dynamic, which means it changes according to position and its dynamism. Manually adaptable contempt of time before NNUE is out of meaning since eval comes from nets anyhow.

Making SF play in more contemptous setting against weaker opponents of course would bring more points against those and produce shorter games against them now and then, but that had to be changed manually then against stronger ones again and so you couldn't let the same setting play as one engine in one tournament, adapting it manually according to opponents wouldn't be fair regards

[Frank Quisinsky]

Hi Peter,

fast draws with white pieces.
The different between Stockfish and Komodo ...

Komodo on rank 1
Stockfish on rank 41

From TOP-41 chess enignes!
In the still running FCP Tourney-2022 after 320 games per engine (Round 08).

An easy statistic with move-average of white-draw games.
A statistic from games, ended with 80 moves!!

Shortly what I mean ...



Best
Frank

And for that reason, Stockfish is for opening book projects not really important!
A random-sample ... of course I have to look in my statistics to the end of the tournament, after 2.000 games per engine!

[peter]

An easy statistic with move-average of white-draw games.
A statistic from games, ended with 80 moves!!

Shortly what I mean ...

This part I did understand, but not the one about the contempt. You would like SF to play more contemptous was what I got, but you would have to change that than from match to match according to the opponent, otherwise you would earn more points against weaker opponents and get more of your beloved short games, but lose some against the stronger ones on the other hand or at least would miss some full points overestimating the positions against the strongest opponents not playing with too high contempt, wouldn't you?
It's a very old discussion and a well known problem, SF's dynamic contempt of nowadays will be best one Elo- wise overall probably, won't that be tested already quite well by SF developers regards

[Frank Quisinsky]

Hi Peter,

Ah, I understand ...

Maybe ...
Stockfish have a bit more aggressiveness as Komodo in the beginning of games after opening book moves.
The price for Stockfish is to high.

If Komodo can solve the problem (vs. "weaker" engines), why Stockfish can't solve the problem?
The differernt between both programs is maybe 15-40 Elo.

I am thinking that it have to do with "Contempt".

The "draw-games" move-average for "white-pieces" is very important for analyzing openings!!

What Stockfish do vs. the others in eng-eng matches is not very important for me.
I like Stockfish but it's not one of my TOP-10 favorite engines for different reasons!

For me is important to find out 10 chess programs, can give me perfect analyzes for opening book positions (next FEOBOS project I will start).
During the FCP Tourney-2022 is still running I will create some statistics. I have to build the perfect TOP-10 engine-team for FEOBOS.

Best
Frank

[amanjpro]

UCI has support of sending Opponent Elo when starting a new game, in a perfect world engines choose their contempt factor based on that, but this is also not that easy to get right

[peter]

If Komodo can solve the problem (vs. "weaker" engines), why Stockfish can't solve the problem?
The differernt between both programs is maybe 15-40 Elo.

I am thinking that it have to do with "Contempt".

If it had to do with contempt (and of course it won't be contempt only making the difference between SF and komodo) that would mean, the 15-40 Elo (if I take your numbers for granted and as for the whole truth, which it isn't neither of course depending on opponents, openings and hardware- time) but than your 15-40 Elo difference would just come from too high a contempt of komodo playing with SF in same match, if this high contempt was the (only one) reason for the so-called solved problem against weaker engines, wouldn't it regards

[carldaman]

Peter, are you saying SF now has internal dynamic contempt? I no longer see a visible and adjustable contempt setting for it.

Regardless, I have noticed the same problem that Frank is describing, as SF simply allows too many draws to significantly weaker engines (lower rated by more than one class).