Komodo 5 64-bit for CCRL 40/4

[Adam Hair]

My only reason for posting this is to give additional data for Komodo 5. My testing is at a shorter time control and is missing a few opponents as compared to Ingo's testing. Also, this is being done on a non-SSE 4.2 cpu (E8400). It may or may not be missing a little Elo boost. And finally, this only includes 119 games at the moment.

CPU : Intel E8400
OS : Windows XP 64-bit
Time Control : 40 moves in 3 minutes, repeating (Conforms to CCRL 40/4 reference)
Openings : a pgn of ~18,000 games, truncated to 4 full moves. Positions are not repeated
CPUs : 1
Hash : 128 MB

The opponents (at the moment) are:
Houdini 2.0c, Rybka 4.1, Stockfish 2.2.2, Critter 1.4, Naum 4.2, Gull 1.0a,
Spike 1.4, Spark 1.0, and Hannibal 1.2.

All opponents are 64-bit engines except Spike.

The following ratings are approximately how they would appear on the CCRL 40/4 Single CPU Pure List. I am showing only the top 16 engines, but the ratings were computed using the entire database for that list. I am going to give the ratings based on how the CCRL calculates ratings now, how the CCRL may calculate the ratings in the future, and with a beta version of Ordo that can calculate and use White advantage.

Bayeselo with default values:

Code: Select all

Rank Name                                 Elo    +    - games score oppo. draws 
   1 Houdini 2.0c 64-bit                 3205   21   21   883   70%  3050   32% 
   2 Komodo 5 64-bit                     3174   57   54   119   69%  3038   30% 
   3 Strelka 5.1 64-bit                  3153   25   25   545   66%  3037   43% 
   4 Critter 1.4 64-bit                  3144   19   19   976   67%  3019   40% 
   5 Komodo 4 64-bit                     3132   17   17  1395   69%  2985   30% 
   6 Stockfish 2.2.2 64-bit              3128   21   20   806   60%  3057   39% 
   7 Rybka 4.1 64-bit                    3124   17   17  1337   67%  2999   35% 
   8 Fritz 13                            2990   18   17  1284   63%  2886   30% 
   9 Naum 4.2 64-bit                     2982   17   17  1155   51%  2975   36% 
  10 Chiron 1.1a 64-bit                  2980   20   20   890   62%  2893   32% 
  11 Deep Shredder 12 64-bit OA On 1CPU  2965   20   19   907   57%  2920   37% 
  12 Hannibal 1.2 64-bit                 2948   22   22   687   51%  2940   36% 
  13 Hiarcs 13.2                         2929   22   22   714   52%  2920   35% 
  14 Gull 1.0a 64-bit                    2928   18   18  1323   65%  2813   27% 
  15 Spark 1.0 64-bit                    2921   16   15  1500   52%  2911   34% 
  16 Spike 1.4 Leiden                    2919   18   18  1178   52%  2910   33% 

Bayeselo using the commands 'elo', 'prior 0.1', 'mm 1 1', and 'scale 1':

Code: Select all

Rank Name                                 Elo    +    - games score oppo. draws 
   1 Houdini 2.0c 64-bit                 3268   22   22   883   70%  3103   32% 
   2 Komodo 5 64-bit                     3236   62   59   119   69%  3089   30% 
   3 Strelka 5.1 64-bit                  3213   27   26   545   66%  3088   43% 
   4 Critter 1.4 64-bit                  3203   21   20   976   67%  3069   40% 
   5 Komodo 4 64-bit                     3190   18   18  1395   69%  3033   30% 
   6 Stockfish 2.2.2 64-bit              3185   22   22   806   60%  3109   39% 
   7 Rybka 4.1 64-bit                    3181   18   18  1337   67%  3048   35% 
   8 Fritz 13                            3038   19   19  1284   63%  2927   30% 
   9 Naum 4.2 64-bit                     3030   19   19  1155   51%  3023   36% 
  10 Chiron 1.1a 64-bit                  3028   22   22   890   62%  2934   32% 
  11 Deep Shredder 12 64-bit OA On 1CPU  3012   21   21   907   57%  2964   37% 
  12 Hannibal 1.2 64-bit                 2993   24   24   687   51%  2985   36% 
  13 Hiarcs 13.2                         2973   24   24   714   52%  2964   35% 
  14 Gull 1.0a 64-bit                    2972   19   19  1323   65%  2850   27% 
  15 Spark 1.0 64-bit                    2965   17   17  1500   52%  2954   34% 
  16 Spike 1.4 Leiden                    2962   19   19  1178   52%  2953   33% 

Ordo

Code: Select all

                            ENGINE   RATING    POINTS  PLAYED     (%) 
               Houdini 2.0c 64-bit   3271.3    622.5       883 70.50%
                   Komodo 5 64-bit   3242.7       82       119 68.90%
                Strelka 5.1 64-bit   3217.9      361       545 66.20%
                Critter 1.4 64-bit   3211.1      654       976 67.00%
                   Komodo 4 64-bit   3190.2    962.5      1395 69.00%
            Stockfish 2.2.2 64-bit   3189.6      482       806 59.80%
                  Rybka 4.1 64-bit   3186.8      894      1337 66.90%
                          Fritz 13   3032.5    812.5      1284 63.30%
                   Naum 4.2 64-bit   3025.4    588.5      1155 51.00%
                Chiron 1.1a 64-bit   3022.6      552       890 62.00%
Deep Shredder 12 64-bit OA On 1CPU   3006.9    512.5       907 56.50%
               Hannibal 1.2 64-bit   2988.5    353.5       687 51.50%
                       Hiarcs 13.2   2969.8      370       714 51.80%
                  Gull 1.0a 64-bit   2967.8      854      1323 64.60%
                  Spark 1.0 64-bit   2959.9    779.5      1500 52.00%
                  Spike 1.4 Leiden   2955.3      607      1178 51.50%

I will update this in 6 to 8 hours.

[Adam Hair]

Update:

Code: Select all

Komodo 5 64-bit               : 199 (+103,= 61,- 35), 67.1 %

Rybka 4.1 64-bit              :  24 (+  7,= 13,-  4), 56.2 %
Hannibal 1.2 64-bit           :  24 (+ 16,=  7,-  1), 81.2 %
Houdini 2.0c 64-bit           :  24 (+  4,=  5,- 15), 27.1 %
Naum 4.2 64-bit               :  24 (+ 19,=  4,-  1), 87.5 %
Critter 1.4 64-bit            :  23 (+  8,= 10,-  5), 56.5 %
Spike 1.4 Leiden              :  20 (+ 14,=  4,-  2), 80.0 %
Spark 1.0 64-bit              :  20 (+ 14,=  5,-  1), 82.5 %
Stockfish 2.2.2 64-bit        :  20 (+  6,=  9,-  5), 52.5 %
Gull 1.0a 64-bit              :  20 (+ 15,=  4,-  1), 85.0 %

So far, Komodo is having trouble with Houdini. But, this is not at odds with what Larry has been saying.

Bayeselo with default values:

Code: Select all

Rank Name                                 Elo    +    - games score oppo. draws 
   1 Houdini 2.0c 64-bit                 3207   21   20   892   71%  3051   32% 
   2 Komodo 5 64-bit                     3162   43   42   199   67%  3037   31% 
   3 Strelka 5.1 64-bit                  3153   25   25   545   66%  3037   43% 
   4 Critter 1.4 64-bit                  3145   19   19   987   67%  3020   40% 
   5 Komodo 4 64-bit                     3132   17   17  1395   69%  2985   30% 
   6 Stockfish 2.2.2 64-bit              3128   20   20   814   60%  3057   39% 
   7 Rybka 4.1 64-bit                    3124   17   16  1345   67%  3000   35% 
   8 Fritz 13                            2990   18   17  1284   63%  2886   30% 
   9 Naum 4.2 64-bit                     2981   17   17  1167   51%  2977   35% 
  10 Chiron 1.1a 64-bit                  2980   20   20   890   62%  2892   32% 
  11 Deep Shredder 12 64-bit OA On 1CPU  2965   20   19   907   57%  2920   37% 
  12 Hannibal 1.2 64-bit                 2947   22   22   695   51%  2942   36% 
  13 Hiarcs 13.2                         2929   22   22   714   52%  2920   35% 
  14 Gull 1.0a 64-bit                    2928   18   18  1331   64%  2815   27% 
  15 Spark 1.0 64-bit                    2920   15   15  1508   52%  2912   34% 
  16 Spike 1.4 Leiden                    2919   18   18  1186   51%  2911   33% 

Bayeselo using the commands 'elo', 'prior', 'mm 1 1', and 'scale 1'.

Code: Select all

Rank Name                                 Elo    +    - games score oppo. draws 
   1 Houdini 2.0c 64-bit                 3270   22   22   892   71%  3104   32% 
   2 Komodo 5 64-bit                     3223   47   45   199   67%  3089   31% 
   3 Strelka 5.1 64-bit                  3213   27   27   545   66%  3088   43% 
   4 Critter 1.4 64-bit                  3204   21   20   987   67%  3070   40% 
   5 Komodo 4 64-bit                     3190   18   18  1395   69%  3033   30% 
   6 Stockfish 2.2.2 64-bit              3186   22   22   814   60%  3110   39% 
   7 Rybka 4.1 64-bit                    3181   18   18  1345   67%  3049   35% 
   8 Fritz 13                            3038   19   19  1284   63%  2927   30% 
   9 Naum 4.2 64-bit                     3029   19   19  1167   51%  3024   35% 
  10 Chiron 1.1a 64-bit                  3027   22   22   890   62%  2934   32% 
  11 Deep Shredder 12 64-bit OA On 1CPU  3012   21   21   907   57%  2963   37% 
  12 Hannibal 1.2 64-bit                 2992   24   24   695   51%  2987   36% 
  13 Hiarcs 13.2                         2973   24   24   714   52%  2963   35% 
  14 Gull 1.0a 64-bit                    2972   19   19  1331   64%  2852   27% 
  15 Spark 1.0 64-bit                    2964   17   17  1508   52%  2955   34% 
  16 Spike 1.4 Leiden                    2962   19   19  1186   51%  2954   33% 

Ordo with White advantage:

Code: Select all

                        ENGINE:  RATING    POINTS  PLAYED    (%)
               Houdini 2.0c 64-bit:  3273.3     629.5     892   70.6%
                   Komodo 5 64-bit:  3226.6     133.5     199   67.1%
                Strelka 5.1 64-bit:  3218.4     361.0     545   66.2%
                Critter 1.4 64-bit:  3212.0     660.0     987   66.9%
            Stockfish 2.2.2 64-bit:  3190.8     486.5     814   59.8%
                   Komodo 4 64-bit:  3190.6     962.5    1395   69.0%
                  Rybka 4.1 64-bit:  3187.0     897.5    1345   66.7%
                          Fritz 13:  3032.6     812.5    1284   63.3%
                   Naum 4.2 64-bit:  3024.9     590.5    1167   50.6%
                Chiron 1.1a 64-bit:  3022.6     552.0     890   62.0%
Deep Shredder 12 64-bit OA On 1CPU:  3006.9     512.5     907   56.5%
               Hannibal 1.2 64-bit:  2988.2     355.0     695   51.1%
                       Hiarcs 13.2:  2969.8     370.0     714   51.8%
                  Gull 1.0a 64-bit:  2967.8     855.5    1331   64.3%
                  Spark 1.0 64-bit:  2959.4     780.0    1508   51.7%
                  Spike 1.4 Leiden:  2955.8     609.0    1186   51.3%

[lkaufman]

Update:

Code: Select all

Komodo 5 64-bit               : 199 (+103,= 61,- 35), 67.1 %

Rybka 4.1 64-bit              :  24 (+  7,= 13,-  4), 56.2 %
Hannibal 1.2 64-bit           :  24 (+ 16,=  7,-  1), 81.2 %
Houdini 2.0c 64-bit           :  24 (+  4,=  5,- 15), 27.1 %
Naum 4.2 64-bit               :  24 (+ 19,=  4,-  1), 87.5 %
Critter 1.4 64-bit            :  23 (+  8,= 10,-  5), 56.5 %
Spike 1.4 Leiden              :  20 (+ 14,=  4,-  2), 80.0 %
Spark 1.0 64-bit              :  20 (+ 14,=  5,-  1), 82.5 %
Stockfish 2.2.2 64-bit        :  20 (+  6,=  9,-  5), 52.5 %
Gull 1.0a 64-bit              :  20 (+ 15,=  4,-  1), 85.0 %

So far, Komodo is having trouble with Houdini. But, this is not at odds with what Larry has been saying.

Bayeselo with default values:

Code: Select all

Rank Name                                 Elo    +    - games score oppo. draws 
   1 Houdini 2.0c 64-bit                 3207   21   20   892   71%  3051   32% 
   2 Komodo 5 64-bit                     3162   43   42   199   67%  3037   31% 
   3 Strelka 5.1 64-bit                  3153   25   25   545   66%  3037   43% 
   4 Critter 1.4 64-bit                  3145   19   19   987   67%  3020   40% 
   5 Komodo 4 64-bit                     3132   17   17  1395   69%  2985   30% 
   6 Stockfish 2.2.2 64-bit              3128   20   20   814   60%  3057   39% 
   7 Rybka 4.1 64-bit                    3124   17   16  1345   67%  3000   35% 
   8 Fritz 13                            2990   18   17  1284   63%  2886   30% 
   9 Naum 4.2 64-bit                     2981   17   17  1167   51%  2977   35% 
  10 Chiron 1.1a 64-bit                  2980   20   20   890   62%  2892   32% 
  11 Deep Shredder 12 64-bit OA On 1CPU  2965   20   19   907   57%  2920   37% 
  12 Hannibal 1.2 64-bit                 2947   22   22   695   51%  2942   36% 
  13 Hiarcs 13.2                         2929   22   22   714   52%  2920   35% 
  14 Gull 1.0a 64-bit                    2928   18   18  1331   64%  2815   27% 
  15 Spark 1.0 64-bit                    2920   15   15  1508   52%  2912   34% 
  16 Spike 1.4 Leiden                    2919   18   18  1186   51%  2911   33% 

Bayeselo using the commands 'elo', 'prior', 'mm 1 1', and 'scale 1'.

Code: Select all

Rank Name                                 Elo    +    - games score oppo. draws 
   1 Houdini 2.0c 64-bit                 3270   22   22   892   71%  3104   32% 
   2 Komodo 5 64-bit                     3223   47   45   199   67%  3089   31% 
   3 Strelka 5.1 64-bit                  3213   27   27   545   66%  3088   43% 
   4 Critter 1.4 64-bit                  3204   21   20   987   67%  3070   40% 
   5 Komodo 4 64-bit                     3190   18   18  1395   69%  3033   30% 
   6 Stockfish 2.2.2 64-bit              3186   22   22   814   60%  3110   39% 
   7 Rybka 4.1 64-bit                    3181   18   18  1345   67%  3049   35% 
   8 Fritz 13                            3038   19   19  1284   63%  2927   30% 
   9 Naum 4.2 64-bit                     3029   19   19  1167   51%  3024   35% 
  10 Chiron 1.1a 64-bit                  3027   22   22   890   62%  2934   32% 
  11 Deep Shredder 12 64-bit OA On 1CPU  3012   21   21   907   57%  2963   37% 
  12 Hannibal 1.2 64-bit                 2992   24   24   695   51%  2987   36% 
  13 Hiarcs 13.2                         2973   24   24   714   52%  2963   35% 
  14 Gull 1.0a 64-bit                    2972   19   19  1331   64%  2852   27% 
  15 Spark 1.0 64-bit                    2964   17   17  1508   52%  2955   34% 
  16 Spike 1.4 Leiden                    2962   19   19  1186   51%  2954   33% 

Ordo with White advantage:

Code: Select all

                        ENGINE:  RATING    POINTS  PLAYED    (%)
               Houdini 2.0c 64-bit:  3273.3     629.5     892   70.6%
                   Komodo 5 64-bit:  3226.6     133.5     199   67.1%
                Strelka 5.1 64-bit:  3218.4     361.0     545   66.2%
                Critter 1.4 64-bit:  3212.0     660.0     987   66.9%
            Stockfish 2.2.2 64-bit:  3190.8     486.5     814   59.8%
                   Komodo 4 64-bit:  3190.6     962.5    1395   69.0%
                  Rybka 4.1 64-bit:  3187.0     897.5    1345   66.7%
                          Fritz 13:  3032.6     812.5    1284   63.3%
                   Naum 4.2 64-bit:  3024.9     590.5    1167   50.6%
                Chiron 1.1a 64-bit:  3022.6     552.0     890   62.0%
Deep Shredder 12 64-bit OA On 1CPU:  3006.9     512.5     907   56.5%
               Hannibal 1.2 64-bit:  2988.2     355.0     695   51.1%
                       Hiarcs 13.2:  2969.8     370.0     714   51.8%
                  Gull 1.0a 64-bit:  2967.8     855.5    1331   64.3%
                  Spark 1.0 64-bit:  2959.4     780.0    1508   51.7%
                  Spike 1.4 Leiden:  2955.8     609.0    1186   51.3%

So, so far you are showing a again over K4 of 30,33, or 36 elo depending on the method of rating. We are currently showing 35. That's pretty good agreement!
It now appears that if there is any problem with AMD, it is small and perhaps not easily addressed by us, since we don't have an AMD machine. So I think it's likely that the initial release is fine for testing on both AMD and Intel.

[MM]

Update:

Code: Select all

Komodo 5 64-bit               : 199 (+103,= 61,- 35), 67.1 %

Rybka 4.1 64-bit              :  24 (+  7,= 13,-  4), 56.2 %
Hannibal 1.2 64-bit           :  24 (+ 16,=  7,-  1), 81.2 %
Houdini 2.0c 64-bit           :  24 (+  4,=  5,- 15), 27.1 %
Naum 4.2 64-bit               :  24 (+ 19,=  4,-  1), 87.5 %
Critter 1.4 64-bit            :  23 (+  8,= 10,-  5), 56.5 %
Spike 1.4 Leiden              :  20 (+ 14,=  4,-  2), 80.0 %
Spark 1.0 64-bit              :  20 (+ 14,=  5,-  1), 82.5 %
Stockfish 2.2.2 64-bit        :  20 (+  6,=  9,-  5), 52.5 %
Gull 1.0a 64-bit              :  20 (+ 15,=  4,-  1), 85.0 %

So far, Komodo is having trouble with Houdini. But, this is not at odds with what Larry has been saying.

Bayeselo with default values:

Code: Select all

Rank Name                                 Elo    +    - games score oppo. draws 
   1 Houdini 2.0c 64-bit                 3207   21   20   892   71%  3051   32% 
   2 Komodo 5 64-bit                     3162   43   42   199   67%  3037   31% 
   3 Strelka 5.1 64-bit                  3153   25   25   545   66%  3037   43% 
   4 Critter 1.4 64-bit                  3145   19   19   987   67%  3020   40% 
   5 Komodo 4 64-bit                     3132   17   17  1395   69%  2985   30% 
   6 Stockfish 2.2.2 64-bit              3128   20   20   814   60%  3057   39% 
   7 Rybka 4.1 64-bit                    3124   17   16  1345   67%  3000   35% 
   8 Fritz 13                            2990   18   17  1284   63%  2886   30% 
   9 Naum 4.2 64-bit                     2981   17   17  1167   51%  2977   35% 
  10 Chiron 1.1a 64-bit                  2980   20   20   890   62%  2892   32% 
  11 Deep Shredder 12 64-bit OA On 1CPU  2965   20   19   907   57%  2920   37% 
  12 Hannibal 1.2 64-bit                 2947   22   22   695   51%  2942   36% 
  13 Hiarcs 13.2                         2929   22   22   714   52%  2920   35% 
  14 Gull 1.0a 64-bit                    2928   18   18  1331   64%  2815   27% 
  15 Spark 1.0 64-bit                    2920   15   15  1508   52%  2912   34% 
  16 Spike 1.4 Leiden                    2919   18   18  1186   51%  2911   33% 

Bayeselo using the commands 'elo', 'prior', 'mm 1 1', and 'scale 1'.

Code: Select all

Rank Name                                 Elo    +    - games score oppo. draws 
   1 Houdini 2.0c 64-bit                 3270   22   22   892   71%  3104   32% 
   2 Komodo 5 64-bit                     3223   47   45   199   67%  3089   31% 
   3 Strelka 5.1 64-bit                  3213   27   27   545   66%  3088   43% 
   4 Critter 1.4 64-bit                  3204   21   20   987   67%  3070   40% 
   5 Komodo 4 64-bit                     3190   18   18  1395   69%  3033   30% 
   6 Stockfish 2.2.2 64-bit              3186   22   22   814   60%  3110   39% 
   7 Rybka 4.1 64-bit                    3181   18   18  1345   67%  3049   35% 
   8 Fritz 13                            3038   19   19  1284   63%  2927   30% 
   9 Naum 4.2 64-bit                     3029   19   19  1167   51%  3024   35% 
  10 Chiron 1.1a 64-bit                  3027   22   22   890   62%  2934   32% 
  11 Deep Shredder 12 64-bit OA On 1CPU  3012   21   21   907   57%  2963   37% 
  12 Hannibal 1.2 64-bit                 2992   24   24   695   51%  2987   36% 
  13 Hiarcs 13.2                         2973   24   24   714   52%  2963   35% 
  14 Gull 1.0a 64-bit                    2972   19   19  1331   64%  2852   27% 
  15 Spark 1.0 64-bit                    2964   17   17  1508   52%  2955   34% 
  16 Spike 1.4 Leiden                    2962   19   19  1186   51%  2954   33% 

Ordo with White advantage:

Code: Select all

                        ENGINE:  RATING    POINTS  PLAYED    (%)
               Houdini 2.0c 64-bit:  3273.3     629.5     892   70.6%
                   Komodo 5 64-bit:  3226.6     133.5     199   67.1%
                Strelka 5.1 64-bit:  3218.4     361.0     545   66.2%
                Critter 1.4 64-bit:  3212.0     660.0     987   66.9%
            Stockfish 2.2.2 64-bit:  3190.8     486.5     814   59.8%
                   Komodo 4 64-bit:  3190.6     962.5    1395   69.0%
                  Rybka 4.1 64-bit:  3187.0     897.5    1345   66.7%
                          Fritz 13:  3032.6     812.5    1284   63.3%
                   Naum 4.2 64-bit:  3024.9     590.5    1167   50.6%
                Chiron 1.1a 64-bit:  3022.6     552.0     890   62.0%
Deep Shredder 12 64-bit OA On 1CPU:  3006.9     512.5     907   56.5%
               Hannibal 1.2 64-bit:  2988.2     355.0     695   51.1%
                       Hiarcs 13.2:  2969.8     370.0     714   51.8%
                  Gull 1.0a 64-bit:  2967.8     855.5    1331   64.3%
                  Spark 1.0 64-bit:  2959.4     780.0    1508   51.7%
                  Spike 1.4 Leiden:  2955.8     609.0    1186   51.3%

So, so far you are showing a again over K4 of 30,33, or 36 elo depending on the method of rating. We are currently showing 35. That's pretty good agreement!
It now appears that if there is any problem with AMD, it is small and perhaps not easily addressed by us, since we don't have an AMD machine. So I think it's likely that the initial release is fine for testing on both AMD and Intel.

Only 199 games. Really few. I would wait at least 1000 games.

Best Regards

[geots]

My only reason for posting this is to give additional data for Komodo 5. My testing is at a shorter time control and is missing a few opponents as compared to Ingo's testing. Also, this is being done on a non-SSE 4.2 cpu (E8400). It may or may not be missing a little Elo boost. And finally, this only includes 119 games at the moment.

CPU : Intel E8400
OS : Windows XP 64-bit
Time Control : 40 moves in 3 minutes, repeating (Conforms to CCRL 40/4 reference)
Openings : a pgn of ~18,000 games, truncated to 4 full moves. Positions are not repeated
CPUs : 1
Hash : 128 MB

The opponents (at the moment) are:
Houdini 2.0c, Rybka 4.1, Stockfish 2.2.2, Critter 1.4, Naum 4.2, Gull 1.0a,
Spike 1.4, Spark 1.0, and Hannibal 1.2.

All opponents are 64-bit engines except Spike.

The following ratings are approximately how they would appear on the CCRL 40/4 Single CPU Pure List. I am showing only the top 16 engines, but the ratings were computed using the entire database for that list. I am going to give the ratings based on how the CCRL calculates ratings now, how the CCRL may calculate the ratings in the future, and with a beta version of Ordo that can calculate and use White advantage.

Bayeselo with default values:

Code: Select all

Rank Name                                 Elo    +    - games score oppo. draws 
   1 Houdini 2.0c 64-bit                 3205   21   21   883   70%  3050   32% 
   2 Komodo 5 64-bit                     3174   57   54   119   69%  3038   30% 
   3 Strelka 5.1 64-bit                  3153   25   25   545   66%  3037   43% 
   4 Critter 1.4 64-bit                  3144   19   19   976   67%  3019   40% 
   5 Komodo 4 64-bit                     3132   17   17  1395   69%  2985   30% 
   6 Stockfish 2.2.2 64-bit              3128   21   20   806   60%  3057   39% 
   7 Rybka 4.1 64-bit                    3124   17   17  1337   67%  2999   35% 
   8 Fritz 13                            2990   18   17  1284   63%  2886   30% 
   9 Naum 4.2 64-bit                     2982   17   17  1155   51%  2975   36% 
  10 Chiron 1.1a 64-bit                  2980   20   20   890   62%  2893   32% 
  11 Deep Shredder 12 64-bit OA On 1CPU  2965   20   19   907   57%  2920   37% 
  12 Hannibal 1.2 64-bit                 2948   22   22   687   51%  2940   36% 
  13 Hiarcs 13.2                         2929   22   22   714   52%  2920   35% 
  14 Gull 1.0a 64-bit                    2928   18   18  1323   65%  2813   27% 
  15 Spark 1.0 64-bit                    2921   16   15  1500   52%  2911   34% 
  16 Spike 1.4 Leiden                    2919   18   18  1178   52%  2910   33% 

Bayeselo using the commands 'elo', 'prior 0.1', 'mm 1 1', and 'scale 1':

Code: Select all

Rank Name                                 Elo    +    - games score oppo. draws 
   1 Houdini 2.0c 64-bit                 3268   22   22   883   70%  3103   32% 
   2 Komodo 5 64-bit                     3236   62   59   119   69%  3089   30% 
   3 Strelka 5.1 64-bit                  3213   27   26   545   66%  3088   43% 
   4 Critter 1.4 64-bit                  3203   21   20   976   67%  3069   40% 
   5 Komodo 4 64-bit                     3190   18   18  1395   69%  3033   30% 
   6 Stockfish 2.2.2 64-bit              3185   22   22   806   60%  3109   39% 
   7 Rybka 4.1 64-bit                    3181   18   18  1337   67%  3048   35% 
   8 Fritz 13                            3038   19   19  1284   63%  2927   30% 
   9 Naum 4.2 64-bit                     3030   19   19  1155   51%  3023   36% 
  10 Chiron 1.1a 64-bit                  3028   22   22   890   62%  2934   32% 
  11 Deep Shredder 12 64-bit OA On 1CPU  3012   21   21   907   57%  2964   37% 
  12 Hannibal 1.2 64-bit                 2993   24   24   687   51%  2985   36% 
  13 Hiarcs 13.2                         2973   24   24   714   52%  2964   35% 
  14 Gull 1.0a 64-bit                    2972   19   19  1323   65%  2850   27% 
  15 Spark 1.0 64-bit                    2965   17   17  1500   52%  2954   34% 
  16 Spike 1.4 Leiden                    2962   19   19  1178   52%  2953   33% 

Ordo

Code: Select all

                            ENGINE   RATING    POINTS  PLAYED     (%) 
               Houdini 2.0c 64-bit   3271.3    622.5       883 70.50%
                   Komodo 5 64-bit   3242.7       82       119 68.90%
                Strelka 5.1 64-bit   3217.9      361       545 66.20%
                Critter 1.4 64-bit   3211.1      654       976 67.00%
                   Komodo 4 64-bit   3190.2    962.5      1395 69.00%
            Stockfish 2.2.2 64-bit   3189.6      482       806 59.80%
                  Rybka 4.1 64-bit   3186.8      894      1337 66.90%
                          Fritz 13   3032.5    812.5      1284 63.30%
                   Naum 4.2 64-bit   3025.4    588.5      1155 51.00%
                Chiron 1.1a 64-bit   3022.6      552       890 62.00%
Deep Shredder 12 64-bit OA On 1CPU   3006.9    512.5       907 56.50%
               Hannibal 1.2 64-bit   2988.5    353.5       687 51.50%
                       Hiarcs 13.2   2969.8      370       714 51.80%
                  Gull 1.0a 64-bit   2967.8      854      1323 64.60%
                  Spark 1.0 64-bit   2959.9    779.5      1500 52.00%
                  Spike 1.4 Leiden   2955.3      607      1178 51.50%

I will update this in 6 to 8 hours.

Adam, this and your other results concerning this test is exceptionally neat and well-done. Tho with you- that is no surprise. A good method choice as well and informative.


Best,

george

[geots]

Look, concerning 40/4 repeating- when you benchmark your system to get your time- I am guessing that in this case, no matter how fast your machine is, you would never run 40/4 any faster than 40/3. Would this be correct?


Thanks,

george

[Adam Hair]

Look, concerning 40/4 repeating- when you benchmark your system to get your time- I am guessing that in this case, no matter how fast your machine is, you would never run 40/4 any faster than 40/3. Would this be correct?


Thanks,

george

Well, if my system was fast enough, I might would consider going 40/2. I would want to make sure that my time control would stay somewhat comparable to the CCRL reference system.

Then again, if it were that fast, I might consider running games for the 40/40 list.

[Adam Hair]

My only reason for posting this is to give additional data for Komodo 5. My testing is at a shorter time control and is missing a few opponents as compared to Ingo's testing. Also, this is being done on a non-SSE 4.2 cpu (E8400). It may or may not be missing a little Elo boost. And finally, this only includes 119 games at the moment.

CPU : Intel E8400
OS : Windows XP 64-bit
Time Control : 40 moves in 3 minutes, repeating (Conforms to CCRL 40/4 reference)
Openings : a pgn of ~18,000 games, truncated to 4 full moves. Positions are not repeated
CPUs : 1
Hash : 128 MB

The opponents (at the moment) are:
Houdini 2.0c, Rybka 4.1, Stockfish 2.2.2, Critter 1.4, Naum 4.2, Gull 1.0a,
Spike 1.4, Spark 1.0, and Hannibal 1.2.

All opponents are 64-bit engines except Spike.

The following ratings are approximately how they would appear on the CCRL 40/4 Single CPU Pure List. I am showing only the top 16 engines, but the ratings were computed using the entire database for that list. I am going to give the ratings based on how the CCRL calculates ratings now, how the CCRL may calculate the ratings in the future, and with a beta version of Ordo that can calculate and use White advantage.

Bayeselo with default values:

Code: Select all

Rank Name                                 Elo    +    - games score oppo. draws 
   1 Houdini 2.0c 64-bit                 3205   21   21   883   70%  3050   32% 
   2 Komodo 5 64-bit                     3174   57   54   119   69%  3038   30% 
   3 Strelka 5.1 64-bit                  3153   25   25   545   66%  3037   43% 
   4 Critter 1.4 64-bit                  3144   19   19   976   67%  3019   40% 
   5 Komodo 4 64-bit                     3132   17   17  1395   69%  2985   30% 
   6 Stockfish 2.2.2 64-bit              3128   21   20   806   60%  3057   39% 
   7 Rybka 4.1 64-bit                    3124   17   17  1337   67%  2999   35% 
   8 Fritz 13                            2990   18   17  1284   63%  2886   30% 
   9 Naum 4.2 64-bit                     2982   17   17  1155   51%  2975   36% 
  10 Chiron 1.1a 64-bit                  2980   20   20   890   62%  2893   32% 
  11 Deep Shredder 12 64-bit OA On 1CPU  2965   20   19   907   57%  2920   37% 
  12 Hannibal 1.2 64-bit                 2948   22   22   687   51%  2940   36% 
  13 Hiarcs 13.2                         2929   22   22   714   52%  2920   35% 
  14 Gull 1.0a 64-bit                    2928   18   18  1323   65%  2813   27% 
  15 Spark 1.0 64-bit                    2921   16   15  1500   52%  2911   34% 
  16 Spike 1.4 Leiden                    2919   18   18  1178   52%  2910   33% 

Bayeselo using the commands 'elo', 'prior 0.1', 'mm 1 1', and 'scale 1':

Code: Select all

Rank Name                                 Elo    +    - games score oppo. draws 
   1 Houdini 2.0c 64-bit                 3268   22   22   883   70%  3103   32% 
   2 Komodo 5 64-bit                     3236   62   59   119   69%  3089   30% 
   3 Strelka 5.1 64-bit                  3213   27   26   545   66%  3088   43% 
   4 Critter 1.4 64-bit                  3203   21   20   976   67%  3069   40% 
   5 Komodo 4 64-bit                     3190   18   18  1395   69%  3033   30% 
   6 Stockfish 2.2.2 64-bit              3185   22   22   806   60%  3109   39% 
   7 Rybka 4.1 64-bit                    3181   18   18  1337   67%  3048   35% 
   8 Fritz 13                            3038   19   19  1284   63%  2927   30% 
   9 Naum 4.2 64-bit                     3030   19   19  1155   51%  3023   36% 
  10 Chiron 1.1a 64-bit                  3028   22   22   890   62%  2934   32% 
  11 Deep Shredder 12 64-bit OA On 1CPU  3012   21   21   907   57%  2964   37% 
  12 Hannibal 1.2 64-bit                 2993   24   24   687   51%  2985   36% 
  13 Hiarcs 13.2                         2973   24   24   714   52%  2964   35% 
  14 Gull 1.0a 64-bit                    2972   19   19  1323   65%  2850   27% 
  15 Spark 1.0 64-bit                    2965   17   17  1500   52%  2954   34% 
  16 Spike 1.4 Leiden                    2962   19   19  1178   52%  2953   33% 

Ordo

Code: Select all

                            ENGINE   RATING    POINTS  PLAYED     (%) 
               Houdini 2.0c 64-bit   3271.3    622.5       883 70.50%
                   Komodo 5 64-bit   3242.7       82       119 68.90%
                Strelka 5.1 64-bit   3217.9      361       545 66.20%
                Critter 1.4 64-bit   3211.1      654       976 67.00%
                   Komodo 4 64-bit   3190.2    962.5      1395 69.00%
            Stockfish 2.2.2 64-bit   3189.6      482       806 59.80%
                  Rybka 4.1 64-bit   3186.8      894      1337 66.90%
                          Fritz 13   3032.5    812.5      1284 63.30%
                   Naum 4.2 64-bit   3025.4    588.5      1155 51.00%
                Chiron 1.1a 64-bit   3022.6      552       890 62.00%
Deep Shredder 12 64-bit OA On 1CPU   3006.9    512.5       907 56.50%
               Hannibal 1.2 64-bit   2988.5    353.5       687 51.50%
                       Hiarcs 13.2   2969.8      370       714 51.80%
                  Gull 1.0a 64-bit   2967.8      854      1323 64.60%
                  Spark 1.0 64-bit   2959.9    779.5      1500 52.00%
                  Spike 1.4 Leiden   2955.3      607      1178 51.50%

I will update this in 6 to 8 hours.

Adam, this and your other results concerning this test is exceptionally neat and well-done. Tho with you- that is no surprise. A good method choice as well and informative.


Best,

george

Thank you, George.

[Adam Hair]

Update:

Code: Select all

Komodo 5 64-bit               : 199 (+103,= 61,- 35), 67.1 %

Rybka 4.1 64-bit              :  24 (+  7,= 13,-  4), 56.2 %
Hannibal 1.2 64-bit           :  24 (+ 16,=  7,-  1), 81.2 %
Houdini 2.0c 64-bit           :  24 (+  4,=  5,- 15), 27.1 %
Naum 4.2 64-bit               :  24 (+ 19,=  4,-  1), 87.5 %
Critter 1.4 64-bit            :  23 (+  8,= 10,-  5), 56.5 %
Spike 1.4 Leiden              :  20 (+ 14,=  4,-  2), 80.0 %
Spark 1.0 64-bit              :  20 (+ 14,=  5,-  1), 82.5 %
Stockfish 2.2.2 64-bit        :  20 (+  6,=  9,-  5), 52.5 %
Gull 1.0a 64-bit              :  20 (+ 15,=  4,-  1), 85.0 %

So far, Komodo is having trouble with Houdini. But, this is not at odds with what Larry has been saying.

Bayeselo with default values:

Code: Select all

Rank Name                                 Elo    +    - games score oppo. draws 
   1 Houdini 2.0c 64-bit                 3207   21   20   892   71%  3051   32% 
   2 Komodo 5 64-bit                     3162   43   42   199   67%  3037   31% 
   3 Strelka 5.1 64-bit                  3153   25   25   545   66%  3037   43% 
   4 Critter 1.4 64-bit                  3145   19   19   987   67%  3020   40% 
   5 Komodo 4 64-bit                     3132   17   17  1395   69%  2985   30% 
   6 Stockfish 2.2.2 64-bit              3128   20   20   814   60%  3057   39% 
   7 Rybka 4.1 64-bit                    3124   17   16  1345   67%  3000   35% 
   8 Fritz 13                            2990   18   17  1284   63%  2886   30% 
   9 Naum 4.2 64-bit                     2981   17   17  1167   51%  2977   35% 
  10 Chiron 1.1a 64-bit                  2980   20   20   890   62%  2892   32% 
  11 Deep Shredder 12 64-bit OA On 1CPU  2965   20   19   907   57%  2920   37% 
  12 Hannibal 1.2 64-bit                 2947   22   22   695   51%  2942   36% 
  13 Hiarcs 13.2                         2929   22   22   714   52%  2920   35% 
  14 Gull 1.0a 64-bit                    2928   18   18  1331   64%  2815   27% 
  15 Spark 1.0 64-bit                    2920   15   15  1508   52%  2912   34% 
  16 Spike 1.4 Leiden                    2919   18   18  1186   51%  2911   33% 

Bayeselo using the commands 'elo', 'prior', 'mm 1 1', and 'scale 1'.

Code: Select all

Rank Name                                 Elo    +    - games score oppo. draws 
   1 Houdini 2.0c 64-bit                 3270   22   22   892   71%  3104   32% 
   2 Komodo 5 64-bit                     3223   47   45   199   67%  3089   31% 
   3 Strelka 5.1 64-bit                  3213   27   27   545   66%  3088   43% 
   4 Critter 1.4 64-bit                  3204   21   20   987   67%  3070   40% 
   5 Komodo 4 64-bit                     3190   18   18  1395   69%  3033   30% 
   6 Stockfish 2.2.2 64-bit              3186   22   22   814   60%  3110   39% 
   7 Rybka 4.1 64-bit                    3181   18   18  1345   67%  3049   35% 
   8 Fritz 13                            3038   19   19  1284   63%  2927   30% 
   9 Naum 4.2 64-bit                     3029   19   19  1167   51%  3024   35% 
  10 Chiron 1.1a 64-bit                  3027   22   22   890   62%  2934   32% 
  11 Deep Shredder 12 64-bit OA On 1CPU  3012   21   21   907   57%  2963   37% 
  12 Hannibal 1.2 64-bit                 2992   24   24   695   51%  2987   36% 
  13 Hiarcs 13.2                         2973   24   24   714   52%  2963   35% 
  14 Gull 1.0a 64-bit                    2972   19   19  1331   64%  2852   27% 
  15 Spark 1.0 64-bit                    2964   17   17  1508   52%  2955   34% 
  16 Spike 1.4 Leiden                    2962   19   19  1186   51%  2954   33% 

Ordo with White advantage:

Code: Select all

                        ENGINE:  RATING    POINTS  PLAYED    (%)
               Houdini 2.0c 64-bit:  3273.3     629.5     892   70.6%
                   Komodo 5 64-bit:  3226.6     133.5     199   67.1%
                Strelka 5.1 64-bit:  3218.4     361.0     545   66.2%
                Critter 1.4 64-bit:  3212.0     660.0     987   66.9%
            Stockfish 2.2.2 64-bit:  3190.8     486.5     814   59.8%
                   Komodo 4 64-bit:  3190.6     962.5    1395   69.0%
                  Rybka 4.1 64-bit:  3187.0     897.5    1345   66.7%
                          Fritz 13:  3032.6     812.5    1284   63.3%
                   Naum 4.2 64-bit:  3024.9     590.5    1167   50.6%
                Chiron 1.1a 64-bit:  3022.6     552.0     890   62.0%
Deep Shredder 12 64-bit OA On 1CPU:  3006.9     512.5     907   56.5%
               Hannibal 1.2 64-bit:  2988.2     355.0     695   51.1%
                       Hiarcs 13.2:  2969.8     370.0     714   51.8%
                  Gull 1.0a 64-bit:  2967.8     855.5    1331   64.3%
                  Spark 1.0 64-bit:  2959.4     780.0    1508   51.7%
                  Spike 1.4 Leiden:  2955.8     609.0    1186   51.3%

So, so far you are showing a again over K4 of 30,33, or 36 elo depending on the method of rating. We are currently showing 35. That's pretty good agreement!
It now appears that if there is any problem with AMD, it is small and perhaps not easily addressed by us, since we don't have an AMD machine. So I think it's likely that the initial release is fine for testing on both AMD and Intel.

My results are still jumping around. I am now showing an improvement over Komodo 4 of 51 to 58 Elo.

Code: Select all

Komodo 5 64-bit               : 298 (+166,= 85,- 47), 70.0 %

Rybka 4.1 64-bit              :  36 (+ 13,= 17,-  6), 59.7 %
Hannibal 1.2 64-bit           :  36 (+ 27,=  8,-  1), 86.1 %
Houdini 2.0c 64-bit           :  34 (+  9,=  7,- 18), 36.8 %
Naum 4.2 64-bit               :  32 (+ 25,=  5,-  2), 85.9 %
Critter 1.4 64-bit            :  32 (+  9,= 16,-  7), 53.1 %
Spike 1.4 Leiden              :  32 (+ 22,=  7,-  3), 79.7 %
Spark 1.0 64-bit              :  32 (+ 24,=  6,-  2), 84.4 %
Stockfish 2.2.2 64-bit        :  32 (+ 12,= 13,-  7), 57.8 %
Gull 1.0a 64-bit              :  32 (+ 25,=  6,-  1), 87.5 %

Bayeselo (default)

Code: Select all

Rank Name                                 Elo    +    - games score oppo. draws 
   1 Houdini 2.0c 64-bit                 3207   21   20   902   70%  3053   31% 
   2 Komodo 5 64-bit                     3183   36   35   298   70%  3034   29% 
   3 Strelka 5.1 64-bit                  3153   25   25   545   66%  3037   43% 
   4 Critter 1.4 64-bit                  3146   19   19   996   67%  3022   40% 
   5 Komodo 4 64-bit                     3132   17   17  1395   69%  2985   30% 
   6 Stockfish 2.2.2 64-bit              3128   20   20   826   59%  3060   39% 
   7 Rybka 4.1 64-bit                    3124   17   16  1357   66%  3002   35% 
   8 Fritz 13                            2990   18   17  1284   63%  2885   30% 
   9 Naum 4.2 64-bit                     2981   17   17  1175   50%  2979   35% 
  10 Chiron 1.1a 64-bit                  2980   20   20   890   62%  2892   32% 
  11 Deep Shredder 12 64-bit OA On 1CPU  2965   20   19   907   57%  2920   37% 
  12 Hannibal 1.2 64-bit                 2946   22   22   707   50%  2946   35% 
  13 Hiarcs 13.2                         2929   22   22   714   52%  2920   35% 
  14 Gull 1.0a 64-bit                    2927   18   18  1343   64%  2818   27% 
  15 Spark 1.0 64-bit                    2920   15   15  1520   51%  2914   34% 
  16 Spike 1.4 Leiden                    2919   18   17  1198   51%  2914   33% 

Bayeselo (prior 0.1, mm 1 1, scale 1)

Code: Select all

Rank Name                                 Elo    +    - games score oppo. draws 
   1 Houdini 2.0c 64-bit                 3270   22   22   902   70%  3106   31% 
   2 Komodo 5 64-bit                     3246   39   38   298   70%  3086   29% 
   3 Strelka 5.1 64-bit                  3213   27   27   545   66%  3088   43% 
   4 Critter 1.4 64-bit                  3205   21   20   996   67%  3072   40% 
   5 Komodo 4 64-bit                     3190   18   18  1395   69%  3033   30% 
   6 Stockfish 2.2.2 64-bit              3186   22   22   826   59%  3113   39% 
   7 Rybka 4.1 64-bit                    3181   18   18  1357   66%  3051   35% 
   8 Fritz 13                            3038   19   19  1284   63%  2927   30% 
   9 Naum 4.2 64-bit                     3029   19   19  1175   50%  3026   35% 
  10 Chiron 1.1a 64-bit                  3027   22   22   890   62%  2934   32% 
  11 Deep Shredder 12 64-bit OA On 1CPU  3012   21   21   907   57%  2963   37% 
  12 Hannibal 1.2 64-bit                 2991   24   24   707   50%  2992   35% 
  13 Hiarcs 13.2                         2973   24   24   714   52%  2963   35% 
  14 Gull 1.0a 64-bit                    2972   19   19  1343   64%  2855   27% 
  15 Spark 1.0 64-bit                    2964   17   17  1520   51%  2958   34% 
  16 Spike 1.4 Leiden                    2963   19   19  1198   51%  2958   33% 

Ordo (with offset)

Code: Select all

                           ENGINE:  RATING    POINTS  PLAYED    (%)
               Houdini 2.0c 64-bit:  3272.9     633.5     902   70.2%
                   Komodo 5 64-bit:  3248.9     208.5     298   70.0%
                Strelka 5.1 64-bit:  3218.5     361.0     545   66.2%
                Critter 1.4 64-bit:  3213.5     665.0     996   66.8%
                   Komodo 4 64-bit:  3190.6     962.5    1395   69.0%
            Stockfish 2.2.2 64-bit:  3190.5     490.5     826   59.4%
                  Rybka 4.1 64-bit:  3186.9     901.5    1357   66.4%
                          Fritz 13:  3032.5     812.5    1284   63.3%
                   Naum 4.2 64-bit:  3025.0     592.0    1175   50.4%
                Chiron 1.1a 64-bit:  3022.6     552.0     890   62.0%
Deep Shredder 12 64-bit OA On 1CPU:  3006.8     512.5     907   56.5%
               Hannibal 1.2 64-bit:  2986.9     355.5     707   50.3%
                       Hiarcs 13.2:  2969.7     370.0     714   51.8%
                  Gull 1.0a 64-bit:  2967.3     856.5    1343   63.8%
                  Spark 1.0 64-bit:  2959.3     781.5    1520   51.4%
                  Spike 1.4 Leiden:  2956.4     611.5    1198   51.0%

[geots]

Look, concerning 40/4 repeating- when you benchmark your system to get your time- I am guessing that in this case, no matter how fast your machine is, you would never run 40/4 any faster than 40/3. Would this be correct?


Thanks,

george

Well, if my system was fast enough, I might would consider going 40/2. I would want to make sure that my time control would stay somewhat comparable to the CCRL reference system.

Then again, if it were that fast, I might consider running games for the 40/40 list.

That is why I asked. I have been running mostly 40/3 repeating for blitz. Not for CCRL reasons- but because there seems to be a lot less chance for time losses with "repeating" blitz. But when I decided to run Komodo 5 vs Houdini 2.0c at 40/40, I decided it would be quite ok to use it benchmarked as CCRL would, which would let me run it at 40/21. Saving me a lot of time and still good 40/40 results. So if I wanted to run blitz also at the same time- it seemed a good idea to take 40/4 and use it benchmarked as CCRL also. But the problem is- that comes out to 40/2 on my machine. I thought I remembered Shaun saying once that no matter how fast it benchmarked here, don't run it faster than 40/3. I may have just dreamed that, it was so long ago. I don't know why I'm thinking he may have said it- but it makes sense sort of, because a fast i7 system might bench at 40/1- and I would think that would be too fast for the reliable results CCRL would want. But maybe not. Maybe I will just run the blitz at 4+2 or 5+3 and then it won't be an issue. And thanks for your answering me.


Best,

george