AH_LTC Match: Houdini 4 vs Komodo 6

[majortom]

Aser has 5 i7-3930K computers that he uses for these matches.

Thank you, Adam!

[lkaufman]

AH_LTC Match: Houdini 4 vs Komodo 6

i7-3930K CPUs 4.25 GHz
90'+30" TC
1 core for all engines
Ponder off
1024 Hash
3-4-5 EGTBs (when available) in SSDs
150 Early Starting Positions Suite, slightly tunned to avoid transpositions (checked with engine vs same-engine matches), and created as a proportional representation of the most played openings/variations on the last recent years in high quality human chess tournaments (source TWIC, only 2400+ ELO players) AH_150_Opening_Suite
All positions are played with Switched Colors for a total of 300 games per match
Games available for download, including eval/time/depth and PV for each move

Results:

Code: Select all

1 Houdini 4  +76/-66/=158 51.67% 155.0/300
2 Komodo 6   +66/-76/=158 48.33% 145.0/300

[/color]

Games: Houdini 4 - Komodo 6

This +12 elo result is very good news for Komodo, because the version now winning TCEC tests as +25 elo over Komodo 6 at bullet chess, so even if this shrinks to say +15 elo at the level used here we would already have passed Houdini 4 at this level. Of course margin of error is much bigger than 3 elo for 300 games, so the issue is not settled yet. But this does support the claim based on TCEC that Komodo (TCEC version) is now the world's strongest engine for serious chess or analysis.

[Aser Huerga]

I'm using 5 twin PCs for these testings: i7-3930K OC to 4.25Ghz CPUs, 16GB, SSDs, same components and configuration.
And HT is off.

Thanks Adam for clarifications

300 are not too much games but at least are much more than in TCEC final (which is a fantastic tournament that I enjoy a lot). Clear difference is that my testings are at 1 core.

I'm running now H4 - SF same dev. version as for the H3 test to see H4 improvement.

Later there will be more testings ...

[shrapnel]

This +12 elo result is very good news for Komodo, because the version now winning TCEC tests as +25 elo over Komodo 6 at bullet chess, so even if this shrinks to say +15 elo at the level used here we would already have passed Houdini 4 at this level. Of course margin of error is much bigger than 3 elo for 300 games, so the issue is not settled yet. But this does support the claim based on TCEC that Komodo (TCEC version) is now the world's strongest engine for serious chess or analysis.

Hi Larry
Don't let Aser Huerga's results get you down ! Komodo 6 is indeed a very strong Engine......with a few parameter tweaks, perhaps stronger than even you think ! (Sorry, but I don't really agree with all your Default parameters ! )
Aser Huerga's results are probably correct if Komodo 6 is using defaults. I'm certainly not using defaults !
In fact, I'm getting such good results, that I've decided NOT to buy Houdini 4, at least for some time !
But I will certainly buy Komodo 7 when it comes out !
Regards

[lkaufman]

This +12 elo result is very good news for Komodo, because the version now winning TCEC tests as +25 elo over Komodo 6 at bullet chess, so even if this shrinks to say +15 elo at the level used here we would already have passed Houdini 4 at this level. Of course margin of error is much bigger than 3 elo for 300 games, so the issue is not settled yet. But this does support the claim based on TCEC that Komodo (TCEC version) is now the world's strongest engine for serious chess or analysis.

Hi Larry
Don't let Aser Huerga's results get you down ! Komodo 6 is indeed a very strong Engine......with a few parameter tweaks, perhaps stronger than even you think ! (Sorry, but I don't really agree with all your Default parameters ! )
Aser Huerga's results are probably correct if Komodo 6 is using defaults. I'm certainly not using defaults !
In fact, I'm getting such good results, that I've decided NOT to buy Houdini 4, at least for some time !
But I will certainly buy Komodo 7 when it comes out !
Regards

Some defaults might be right for blitz but wrong for slow chess. For example table memory size should probably be larger for slow games; for TCEC we used 4x the default. Which defaults do you think are wrong, and for what time limit?

[Aser Huerga]

For the record: I know that some settings could be better for engine X in particular conditions, but I decided to run all my test at default settings because if I change some settings for engine X an author can claim another setting is better for his engine against engine X, and so on ...

Of course there's nothing wrong testing other settings, I think is very interesting, but I choosed to do it this way.

[gotogo]

Have you used 2 PC or 1?
I found 2 names in PGN tags: PC-PC & NITRO-PC.
Did that games played with HT?

It seams weird, there are 300 H4 games played. TC is 90'+30'' which is like 270 minutes on average per game. For 300 games it is 81000 minutes or 1350 hours. Divided by 6 cores it is 225 hours. Even on 2 computers that is still 112.5 hours and H4 appeared 56 hours ago.
So either it was on 4 machines in parallel (24 cores) or do I smell cheating?

Don't forget that many games has played with Arena & user adjudication.
And if the games has played on 2 PC with hyper-threading on (24 threads), we have x2 games for the same time.

Depths achieved from pgn are not impressive, it seams slow for 3930, which on 4.25GHz with H3 should give around 2.6Mnps per core (HT off). In 5minutes initial search depth 24-25 seams low for H3.
So you are right it might be HT with around 3.2Mnps per core (1.6Mnps per thread). But in that case running 12 independent games on 6 cores is suicide and engines just fight for resources and basically the more aggressive one wins, so that kind of test is totally meaningless.
That could also explain pretty bad performance of H3 and H4. There is post on immoratalchess of 5min 16-cores games which is effectively longer TC than this one if HT is on here and there H4 kills K6. So I strongly believe these results are totally bogus.

remember K6 is not the same engine played in tcec match... k6 is a watered down version and the one in tcec... the tcec versions shouldn't be allowed unless it is a public engine. to me the tcec results is total crap and bogus playing engines not offered to public. all other engines except bouquet 1.8b and komodo are public and was with promise to be made public . komodo had a new version each match... this years tcec was total bull....

[Adam Hair]

Have you used 2 PC or 1?
I found 2 names in PGN tags: PC-PC & NITRO-PC.
Did that games played with HT?

It seams weird, there are 300 H4 games played. TC is 90'+30'' which is like 270 minutes on average per game. For 300 games it is 81000 minutes or 1350 hours. Divided by 6 cores it is 225 hours. Even on 2 computers that is still 112.5 hours and H4 appeared 56 hours ago.
So either it was on 4 machines in parallel (24 cores) or do I smell cheating?

Don't forget that many games has played with Arena & user adjudication.
And if the games has played on 2 PC with hyper-threading on (24 threads), we have x2 games for the same time.

Depths achieved from pgn are not impressive, it seams slow for 3930, which on 4.25GHz with H3 should give around 2.6Mnps per core (HT off). In 5minutes initial search depth 24-25 seams low for H3.
So you are right it might be HT with around 3.2Mnps per core (1.6Mnps per thread). But in that case running 12 independent games on 6 cores is suicide and engines just fight for resources and basically the more aggressive one wins, so that kind of test is totally meaningless.
That could also explain pretty bad performance of H3 and H4. There is post on immoratalchess of 5min 16-cores games which is effectively longer TC than this one if HT is on here and there H4 kills K6. So I strongly believe these results are totally bogus.

remember K6 is not the same engine played in tcec match... k6 is a watered down version and the one in tcec... the tcec versions shouldn't be allowed unless it is a public engine. to me the tcec results is total crap and bogus playing engines not offered to public. all other engines except bouquet 1.8b and komodo are public and was with promise to be made public . komodo had a new version each match... this years tcec was total bull....

Really?

[lkaufman]

Have you used 2 PC or 1?
I found 2 names in PGN tags: PC-PC & NITRO-PC.
Did that games played with HT?

It seams weird, there are 300 H4 games played. TC is 90'+30'' which is like 270 minutes on average per game. For 300 games it is 81000 minutes or 1350 hours. Divided by 6 cores it is 225 hours. Even on 2 computers that is still 112.5 hours and H4 appeared 56 hours ago.
So either it was on 4 machines in parallel (24 cores) or do I smell cheating?

Don't forget that many games has played with Arena & user adjudication.
And if the games has played on 2 PC with hyper-threading on (24 threads), we have x2 games for the same time.

Depths achieved from pgn are not impressive, it seams slow for 3930, which on 4.25GHz with H3 should give around 2.6Mnps per core (HT off). In 5minutes initial search depth 24-25 seams low for H3.
So you are right it might be HT with around 3.2Mnps per core (1.6Mnps per thread). But in that case running 12 independent games on 6 cores is suicide and engines just fight for resources and basically the more aggressive one wins, so that kind of test is totally meaningless.
That could also explain pretty bad performance of H3 and H4. There is post on immoratalchess of 5min 16-cores games which is effectively longer TC than this one if HT is on here and there H4 kills K6. So I strongly believe these results are totally bogus.

remember K6 is not the same engine played in tcec match... k6 is a watered down version and the one in tcec... the tcec versions shouldn't be allowed unless it is a public engine. to me the tcec results is total crap and bogus playing engines not offered to public. all other engines except bouquet 1.8b and komodo are public and was with promise to be made public . komodo had a new version each match... this years tcec was total bull....

What is wrong with introducing new versions at each stage, as long as this right applies to all? If anything, the rule was unusually strict, as in normal computer chess tournaments program changes are allowed after EACH GAME. If an engine is not public and does well, you can expect it to be offered for sale soon after; if it does poorly who cares?

[shrapnel]

Some defaults might be right for blitz but wrong for slow chess. For example table memory size should probably be larger for slow games; for TCEC we used 4x the default. Which defaults do you think are wrong, and for what time limit?

Hi Larry
I've sent you a PM.
I'd prefer not to let my Settings be generally known.