Komodo 9 @ CEGT

[Leto]

A new update to the 40/4 list will be coming Monday May 25. I had car issues the past two weeks but I was able to get some matches done for the 12CPU and the 8CPU versions of Komodo 9 while I looked for a new car.

For the 8CPU version of Komodo 9 I have about 1400 games at the moment and the results between my Intel and AMD computers were practically identical.

My apologies, I meant to say tomorrow's update will add 1000 games for the 8CPU version, 600 games for the 12CPU version.

[lkaufman]

So on your hardware Stockfish runs better on AMD, Komodo runs better on Intel, with a fairly significant difference, although this is apparently not a hard and fast rule. Even among my own big Intel machines, some favor Komodo more, some Stockfish, although the differences are smaller than you are observing. A tester could bias the results (intentionally or unintentionally) just by his choice of which computer to use for a certain test. As long as a tester uses all the machines he has on the same test, it won't be an issue. Since I think all the testers are fair, it's probably not much of a practical problem, just something to keep in mind.

[Graham Banks]

.. I meant to say tomorrow's update will add 1000 games for the 8CPU version, 600 games for the 12CPU version.

Great effort Leto.

[ThatsIt]

Our new update for the CEGT 40/4 list is online now:
http://www.husvankempen.de/nunn/blitz.htm

Best wishes,
G.S.
(CEGT team)

[Leto]

So on your hardware Stockfish runs better on AMD, Komodo runs better on Intel, with a fairly significant difference, although this is apparently not a hard and fast rule. Even among my own big Intel machines, some favor Komodo more, some Stockfish, although the differences are smaller than you are observing. A tester could bias the results (intentionally or unintentionally) just by his choice of which computer to use for a certain test. As long as a tester uses all the machines he has on the same test, it won't be an issue. Since I think all the testers are fair, it's probably not much of a practical problem, just something to keep in mind.

On my 8 core testing I run the same matches on both machines. For my Komodo 9 testing I got almost identical results on both machines, here's what I got:

Intel 5620:
K9 vs SF6 8: 53.75%
K9 vs SF5 8: 59%
K9 vs Critter 1.6 4: 80%
K9 vs H4 8: 66%
K9 vs R4.1 8: 86%
K9 vs SF5 4: 61%
K9 vs SF5 1: 70%

AMD FX8300:
K9 vs SF6 8: 52% -1.75%
K9 vs SF5 8: 61% +2%
K9 vs Critter 1.6 4: 83% +3%
K9 vs H4 8: 65% -1%
K9 vs R4.1 8: 79% -7%
K9 vs SF5 4: 62% +1%
K9 vs SF5 1: 71% +1%

In this case K9 8CPU scored a little under 3% higher on the Intel machine than on the AMD machine, but the biggest difference was the match against Rybka 4.1 8CPU where K9 managed to score 3.5 points higher on the Intel machine than the AMD machine, but then again that was just a 50 game match, run it again and the result could be reversed.

Take away the match against Rybka and K9 is scoring 4.25% higher on the AMD machine.

[Leto]

Our new update for the CEGT 40/4 list is online now:
http://www.husvankempen.de/nunn/blitz.htm

Best wishes,
G.S.
(CEGT team)

With K9 just 25 points away from 3400 it's looking likely that an engine will surpass the 3400 barrier this year. Very excited to see that.

[Modern Times]

On my 8 core testing I run the same matches on both machines. For my Komodo 9 testing I got almost identical results on both machines.

In my view, the differences in your NPS ratios and mine are so small that they won't affect any testing results. RELAX Larry

The nps bench figures even vary slightly every time you run the bench command, so on the same machine you could get a 1-2% variance if you took the lower of one and the higher of the other.

Larry, at one point you were considering an AMD specific compile. Did you drop that because you didn't get any improvement from it ?

[lkaufman]

On my 8 core testing I run the same matches on both machines. For my Komodo 9 testing I got almost identical results on both machines.

In my view, the differences in your NPS ratios and mine are so small that they won't affect any testing results. RELAX Larry

The nps bench figures even vary slightly every time you run the bench command, so on the same machine you could get a 1-2% variance if you took the lower of one and the higher of the other.

Larry, at one point you were considering an AMD specific compile. Did you drop that because you didn't get any improvement from it ?

We don't own any AMD machines. We could probably find someone we trust to do an AMD compile, but it would have to be 5% faster to justify a release, which seems a bit unlikely, especially since results vary greatly from one model computer to another even within a family. I'm not very concerned about this issue, it's just something to remember in case we see big discrepancies between testing groups, which is not the case now.