10 min + 5 sec engines ratinglist

[majortom]

Thank you, Ray!

[majortom]

@Ray: I hate to say this, but are you sure about the error bars of your list? I can not run Bayeselo now but it seems that these error bars are for less than 95% confidence. Could you please check it? Thanks in advance. You also do a great job.

Regards from Spain.

Ajedrecista.

There are a lot of draws, that's why error bars is +-10 elo, I suppose.

[majortom]

TC 10'+5"

i7-3960x (overclocked to ~4103 MHZ)
1-core (HT off)
ponder off
colours reversed openings
512 mb hash
GUI cutechess-cli

4 matches

SF 190514 x64 vs. Gull 3 x64
SF 190514 x64 vs. Komodo 7a x64
SF 190514 x64 vs. Houdini Pro 4B x64
Komodo 7a x64 vs. Houdini Pro 4B x64

1000 games for each match (500 8-moves openings set with 441 various ECOs)

[majortom]

After 19 hours:

Code: Select all

1 Stockfish 190514 64 SSE4.2: 3019   99 (+ 26,= 55,- 18), 54.0 %

Houdini 4 Pro x64             :  33 (+  8,= 18,-  7), 51.5 %
Komodo 7a 64-bit              :  32 (+  9,= 19,-  4), 57.8 %
Gull 3 x64                    :  34 (+  9,= 18,-  7), 52.9 %

2 Komodo 7a 64-bit          : 3001   65 (+ 16,= 34,- 15), 50.8 %

Stockfish 190514 64 SSE4.2    :  32 (+  4,= 19,-  9), 42.2 %
Houdini 4 Pro x64             :  33 (+ 12,= 15,-  6), 59.1 %

3 Gull 3 x64                : 2998   34 (+  7,= 18,-  9), 47.1 %

Stockfish 190514 64 SSE4.2    :  34 (+  7,= 18,-  9), 47.1 %

4 Houdini 4 Pro x64         : 2973   66 (+ 13,= 33,- 20), 44.7 %

Stockfish 190514 64 SSE4.2    :  33 (+  7,= 18,-  8), 48.5 %
Komodo 7a 64-bit              :  33 (+  6,= 15,- 12), 40.9 %

[majortom]

132 games played for 19 hours ~34 min 32 sec (the average duration of games of each match played on single core) / 4 (number of used cores) ~8 min 38 sec (the average duration of games of 4 matches - CPU time).

Every 5 min auto-updating files for each match:

SF 190514 x64 vs. Gull 3 x64
SF 190514 x64 vs. Komodo 7a x64
SF 190514 x64 vs. Houdini Pro 4B x64
Komodo 7a x64 vs. Houdini Pro 4B x64

and mixed file (all-in-one):

all_in_one_23052014.pgn

[lkaufman]

TC 10'+5"

i7-3960x (overclocked to ~4103 MHZ)
1-core (HT off)
ponder off
colours reversed openings
512 mb hash
GUI cutechess-cli

4 matches

SF 190514 x64 vs. Gull 3 x64
SF 190514 x64 vs. Komodo 7a x64
SF 190514 x64 vs. Houdini Pro 4B x64
Komodo 7a x64 vs. Houdini Pro 4B x64

1000 games for each match (500 8-moves openings set with 441 various ECOs)

I just want to say that I really look forward to these results, because they will be free of the effects of running on AMD and of playing inferior opposition, and the time limit is quite respectable. If Komodo 7 doesn't do well, I won't have any excuse!

[Vinvin]

TC 10'+5"

i7-3960x (overclocked to ~4103 MHZ)
1-core (HT off)
ponder off
colours reversed openings
512 mb hash
GUI cutechess-cli

4 matches

SF 190514 x64 vs. Gull 3 x64
SF 190514 x64 vs. Komodo 7a x64
SF 190514 x64 vs. Houdini Pro 4B x64
Komodo 7a x64 vs. Houdini Pro 4B x64

1000 games for each match (500 8-moves openings set with 441 various ECOs)

I just want to say that I really look forward to these results, because they will be free of the effects of running on AMD and of playing inferior opposition, and the time limit is quite respectable. If Komodo 7 doesn't do well, I won't have any excuse!

Not bad until now :

Code: Select all

1   Komodo 7a 64-bit   +21/=42/-11 56.76%   42.0/74
2   Houdini 4 Pro x64  +11/=42/-21 43.24%   32.0/74

Code: Select all

1   Stockfish 190514 64 SSE4.2  +19/=43/-13 54.00%   40.5/75
2   Komodo 7a 64-bit            +13/=43/-19 46.00%   34.5/75

[Modern Times]

I just want to say that I really look forward to these results, because they will be free of the effects of running on AMD

I don't think the evidence you've provided on that is conclusive at all. We would need tests under identical conditions, and with thousands of games to tease out any difference.

and of playing inferior opposition, and the time limit is quite respectable. If Komodo 7 doesn't do well, I won't have any excuse!

Fact is, inferior opposition is important. You will come up against them in tournament competition, and your ability to crush them is important. A draw rather than a win could mean the difference between victory and 2nd place. Having said that, for assessing Komodo as an analysis tool, they are not useful.

[lkaufman]

I just want to say that I really look forward to these results, because they will be free of the effects of running on AMD

I don't think the evidence you've provided on that is conclusive at all. We would need tests under identical conditions, and with thousands of games to tease out any difference.

and of playing inferior opposition, and the time limit is quite respectable. If Komodo 7 doesn't do well, I won't have any excuse!

Fact is, inferior opposition is important. You will come up against them in tournament competition, and your ability to crush them is important. A draw rather than a win could mean the difference between victory and 2nd place. Having said that, for assessing Komodo as an analysis tool, they are not useful.

On the amd/intel question, you don't need any games at all, you just need to compare relative NPS.

The second question is the same as whether the world (human) champion should be the winner in a direct match or small RR of the best players, or the one with the highest scores in mixed level tournaments. Almost everyone favors the former.

[Modern Times]

On the amd/intel question, you don't need any games at all, you just need to compare relative NPS.

I disagree totally. We may know that a doubling in speed is worth a certain amount of Elo, but we don't know that a 50% increase is 50% of that Elo, or a 10% increase is 10% of that Elo. There is no evidence of shape the line or curve between those two points.