Number 1 engine on long time controls

[tomgdrums]

What I'm doing here is a study, not marketing. The data shows that Houdini does not scale well. I publish this and my methodology so that it can be duplicated and verified by others. I run 2000 games because we get a lot of nonsense statements from people who run 10 games matches to draw conclusions, so I would invite anyone to duplicate this test with more games.

You don't just report data, you SPECULATE about how good your own engine Komodo might be at some time control you haven't actually used in your test.
In addition you SPECULATE about a commercial competitor, Houdini 2, a program you've never actually used.

As a forum moderator, don't you feel a little bit ethically challenged when writing this marketing blabla disguised as "a study"?

Robert

I'm not going to engage in a pissing contest with you. Make anything out of that you wish. Meanwhile I'll refer you to this long time control rating list:

http://www.husvankempen.de/nunn//40120n ... liste.html

What is interesting about that rating list is that Houdini 2 is nowhere to be found. Seems a tad incomplete to me.

[Don]

What I'm doing here is a study, not marketing. The data shows that Houdini does not scale well. I publish this and my methodology so that it can be duplicated and verified by others. I run 2000 games because we get a lot of nonsense statements from people who run 10 games matches to draw conclusions, so I would invite anyone to duplicate this test with more games.

You don't just report data, you SPECULATE about how good your own engine Komodo might be at some time control you haven't actually used in your test.
In addition you SPECULATE about a commercial competitor, Houdini 2, a program you've never actually used.

As a forum moderator, don't you feel a little bit ethically challenged when writing this marketing blabla disguised as "a study"?

Robert

I'm not going to engage in a pissing contest with you. Make anything out of that you wish. Meanwhile I'll refer you to this long time control rating list:

http://www.husvankempen.de/nunn//40120n ... liste.html

What is interesting about that rating list is that Houdini 2 is nowhere to be found. Seems a tad incomplete to me.

They cannot thoroughly test every version of every program at that time control. I don't think many people are willing to test that long and of course it take a lot of time to get just a few games.

Komodo would do better if it were Houdini 2.0 at this time control. On the CEGT list Houdini 2.0 is about 15 ELO stronger at 40/4 but at 40/20 already Houdini 1.5 is stronger (look at the single processor version:

http://www.husvankempen.de/nunn/40_40%2 ... liste.html

On almost every rating list you can see the trend, Houdini 2.0 is awesome at fast time controls but then Houdini 1.5 gradually catches up and then passes it at the longer time controls.

Another example is CCRL At 40/4 Houdini 2.0 is a whopping 74 ELO ahead of Houdini 1.5 but at 40/40 Houdini 1.5 is +13 ELO stronger!

The IPON list is played at a a level that is significantly slower than the CCRL 40/4 time control (which is adjusted for hardware) and as you would expect Houdini 1.5 does better, in fact it's 7 ELO stronger than Houdini 2.0

The trend is so obvious that I am surprised nobody else has noticed it.

Actually EVERY program has specific scaling characteristics. It drives us crazy when we are testing because we cannot use any program as a reference due this fact. They will either go up, go down, or stay about the same with increasing time control but of course that is always relative to some other program or programs. We test at many different time controls because we know about this.

[Houdini]

Look, you're pissing again... (a contest you're clearly winning!)

If you would have actually tested Houdini 2 instead of inventing stuff to suit your marketing story, you would understand better which of these rating lists results are real, and which are clearly wrong.

Robert

[Peter Skinner]

Look, you're pissing again... (a contest you're clearly winning!)

If you would have actually tested Houdini 2 instead of inventing stuff to suit your marketing story, you would understand better which of these rating lists results are real, and which are clearly wrong.

Robert

I've messaged you for this very fact several times about getting Houdini 2.0 for testing purposes.

I do test games for our internal testing with Crafty, and since I am the only team member that uses Windows that is why I have requested such.

I haven't had a reply to a single PM that I've sent you.

Peter

[FriedmannC]

Hi Uri, I was thinking KOMODO 4 MP is due to be released very soon, while we must wait several months for the upcoming HOUDINI 3 and DEEP RYBKA 5.
Best regards,
Catalin

[Werewolf]

Komodo would do better if it were Houdini 2.0 at this time control. On the CEGT list Houdini 2.0 is about 15 ELO stronger at 40/4 but at 40/20 already Houdini 1.5 is stronger (look at the single processor version:

http://www.husvankempen.de/nunn/40_40%2 ... liste.html

Why do you appeal to the SP version? If the aim is to show Houdini is bad at LTC (or that Komodo is strong at LTC) I would have thought we'd want as many processors involved as possible as it allows us to simulate LTC in less time!

When Komodo MP comes out, for example, surely you'll get to see your theory played out at faster time controls than Komodo SP can manage..?

[geots]

What I'm doing here is a study, not marketing. The data shows that Houdini does not scale well. I publish this and my methodology so that it can be duplicated and verified by others. I run 2000 games because we get a lot of nonsense statements from people who run 10 games matches to draw conclusions, so I would invite anyone to duplicate this test with more games.

You don't just report data, you SPECULATE about how good your own engine Komodo might be at some time control you haven't actually used in your test.
In addition you SPECULATE about a commercial competitor, Houdini 2, a program you've never actually used.

As a forum moderator, don't you feel a little bit ethically challenged when writing this marketing blabla disguised as "a study"?

Robert

I'm not going to engage in a pissing contest with you. Make anything out of that you wish. Meanwhile I'll refer you to this long time control rating list:

http://www.husvankempen.de/nunn//40120n ... liste.html

What is interesting about that rating list is that Houdini 2 is nowhere to be found. Seems a tad incomplete to me.

A lot more than a tad incomplete. Where is Fritz 13- am i to believe Stockfish 2.0 is stronger here than 2 releases later- what the hell is Rybka 1.2f doing here even being tested- it is ancient. No games for Deep Junior 13. Why Rybka 4. Where is 4.1. Not even mentioning that to watch games at this control is worse than sitting on the porch watching paint dry. I would buy absolutely nothing based on this list. When I am shown the PGNs from the top engines here where they defeated in a match RobblLito 0.10, any of a dozen Ivanhoe versions, Houdini 2 Pro, Deep Saros, Vitruvius, Strelka 5.1- then come back to me. I am not interested in hearing about this clone or that clone, or what one should do based on this and that. And by the way- I keep hearing the term "Ippo clones" thrown around. My reaction to that- Show Me the Proof On Paper and Let Me Have It Verified By A 2nd Opinion. Or to put it bluntly- Prove_It.


gts

[jdart]

Just as another data point, Houdini is used by a number of very strong computer accounts on playchess.com - for example handles "Blackborn", "Pastorale", "Ultrapower" - most of these seem to have moved to Houdini 2.0 instead of 1.5, so apparently they think it is superior (doesn't mean they are right, but these accounts are doing well against most other engines).

[acase]

Just as another data point, Houdini is used by a number of very strong computer accounts on playchess.com - for example handles "Blackborn", "Pastorale", "Ultrapower" - most of these seem to have moved to Houdini 2.0 instead of 1.5, so apparently they think it is superior (doesn't mean they are right, but these accounts are doing well against most other engines).

That is true Jon, and those accounts you named mostly play only at long time controls.

[bhlangonijr]

Why do you appeal to the SP version?

Because there's no Komodo MP yet..?

If the aim is to show Houdini is bad at LTC (or that Komodo is strong at LTC) I would have thought we'd want as many processors involved as possible as it allows us to simulate LTC in less time!

Your reasoning is cute, but flawed. Increasing the number of processors doesn't "simulate LTC in less time". There are many reasons: Different implementations of MP, scalability issues, indeterminism, etc.

If for instance a MP version of engine A is much better than MP version of B, it only proves that MP techniques/algorithms of engine A is better than that of engine B. Period.

When Komodo MP comes out, for example, surely you'll get to see your theory played out at faster time controls than Komodo SP can manage..?