Review of Stockfish and Komodo

[Sean Evans]

And Then There Were Two

"The results are fairly clear. Komodo 9 is ever so slightly stronger than Stockfish 6 when it comes to engine-engine play, and this advantage seems to grow when longer time controls are used."

[Graham Banks]

Nice read.
When I got near the bottom, I got a shock, as the photo looked like Martin Thoreson. Then I realised it wasn't!

[schack]

Nah, the picture is of me. Note all the links to discussions here at this very forum! Hope y'all like it.

[Graham Banks]

Nah, the picture is of me. Note all the links to discussions here at this very forum! Hope y'all like it.

Great to see you here, John.

[Paloma]

you used all 4 cores or 1 core for the test ?

[schack]

2 cpus with 2gb RAM each

[Laskos]

2 cpus with 2gb RAM each

That test suite is mostly tactical, and Houdini 4 Tactical performs by far the best there. Closest to matching strength of top engines seems STS (Strategic Test Suite), but Komodo 9 still comes behind Houdini 4 there too (with Stockfish very close to Houdini).

[schack]

I think I misunderstood. For the Talkchess text, I used four cores (not eight, so no HT) with 4gb of ram.

[lkaufman]

2 cpus with 2gb RAM each

That test suite is mostly tactical, and Houdini 4 Tactical performs by far the best there. Closest to matching strength of top engines seems STS (Strategic Test Suite), but Komodo 9 still comes behind Houdini 4 there too (with Stockfish very close to Houdini).

I think the STS is pretty good for measuring progress as an engine improves, but not so good for comparing unrelated engines, especially Houdini. The issue is that I believe STS was verified with Rybka, and so to some extent it is a similarity test with Rybka rather than a pure strength test. Since Houdini started as a copy of Ippolit, and Ippolit was heavily based on Rybka, Houdini is much more similar to Rybka than either Stockfish or Komodo.

[Laskos]

2 cpus with 2gb RAM each

That test suite is mostly tactical, and Houdini 4 Tactical performs by far the best there. Closest to matching strength of top engines seems STS (Strategic Test Suite), but Komodo 9 still comes behind Houdini 4 there too (with Stockfish very close to Houdini).

I think the STS is pretty good for measuring progress as an engine improves, but not so good for comparing unrelated engines, especially Houdini. The issue is that I believe STS was verified with Rybka, and so to some extent it is a similarity test with Rybka rather than a pure strength test. Since Houdini started as a copy of Ippolit, and Ippolit was heavily based on Rybka, Houdini is much more similar to Rybka than either Stockfish or Komodo.

Swami wrote on the STS page "There's a general misconception that STS is solely based on a long hour analysis of Rybka alone". I still believe though that in some cases it's more a similarity test. On the other hand, most other test suites, almost all of them tactical, are much worse predictors of strength. Recently I used the STS tool of Ferdinand Mosca to get the global score (matching not only best solution, but second, third, etc. solutions), and K9, SF, H4 are almost indistinguishable in the total points, with a little peculiarity of K9 of solving a bit less than the other two engines of the best solutions, but more of other, a bit worse solutions. Basically, it would say that K9 makes less blunders of this, more strategical kind, if what I saw was not noise.