Komodo MP vs Houdini 3

[IWB]

ingo, you run a fantastic testing site (my favorite), but why do you no longer test 2 cores? is it a matter of resources, or just time or preference? i feel that'd be an excellent addition, if it's possible, as it'd give us an idea of how well multicore is implemented. in this day & age, of course, everyone has at least a dual core machine.

Personal preference, time, money ... all of those.
I am interested in the enigne performance, not necceseraly in the MP capabilities. For my own use I add 50 Elo for 2 cores, another 35 to 40 for 4 cores, another 20-25 for 8 cores. I might be wrong by 10 or even 20 ELo with this "over the thumb rule" and some engines might change rankings, but I am not in a different ballpark (to use that american expression). For me that estimation is good enough.

i notice among the outliers so far (50 elo from performance elo), komodo's results are pretty evenly distributed between stronger & weaker engines, despite larry's (great, imo) decision to make it a bit more dynamic. this seems like a very good thing, as there's no real ratings inflation against weaker competition.

Komodo 5.1 - Deep Junior 13.3 (2743) 49.0 - 15.0 76.56% Perf=2948
Komodo 5.1 - Quazar 0.4 (2726) 57.0 - 6.0 90.48% Perf=3117

A Performance difference of more than 150 ELo is something I would not call "evenly distributed" ... but so what, it is just 150 games at the end vs each opponent. For me that is not enough to draw such conclusions, I do not care for the individual results and can only warn to do that (just out of 150 games)

again, because houdini defaults with contempt, i wonder if this may've been the case, but i can't find a page showing its test results.

I dissagree and agree with the contempt thing for Houdini (or Rybka). If you want to make a rating list you have to stick with it. If you want to develop a new engine you should do the best to get better opponents than your own engine. So, internaly switch it off, in public leave it in.

Besides that: Every programmer who decides to include that is doing it to gain some points vs weaker engines. Of course it backfires when something stronger is on the market - thats how it goes ...

Bye
Ingo

[yanquis1972]

Komodo 5.1 - Deep Junior 13.3 (2743) 49.0 - 15.0 76.56% Perf=2948
Komodo 5.1 - Quazar 0.4 (2726) 57.0 - 6.0 90.48% Perf=3117

A Performance difference of more than 150 ELo is something I would not call "evenly distributed" ... but so what, it is just 150 games at the end vs each opponent. For me that is not enough to draw such conclusions, I do not care for the individual results and can only warn to do that (just out of 150 games)

from my arbitrary baseline of +/- 50 elo compared to performance elo (thus far). we have:

K5.1: 838.5 - 318.5 72.47% Perf=3008



Komodo 5.1 -Critter 1.4a (2977) 41.5 - 23.5 63.85% Perf=3075 HIGH

Komodo 5.1 - Gull II (2929) 44.5 - 20.5 68.46% Perf=3063 HIGH

Komodo 5.1 - Spike 1.4 32b (2767) 54.0 - 10.0 84.38% Perf=3059 HIGH

Komodo 5.1 - Deep Junior 13.3 (2743) 49.0 - 15.0 76.56% Perf=2948 LOW

Komodo 5.1 - Quazar 0.4 (2726) 58.0 - 6.0 90.63% Perf=3120 VERY HIGH

so, 2 against 'stronger' engines' 3, against 'weaker', but given it's scoring -lower- than expected against an engine i would put in the 'weaker' category (junior), i consider that a fairly even distribution. just an observation i made, could very well mean absolutely nothing, but if it does mean anything, it's a positive imo.

[carldaman]

Fritz, as far as I know, saves all the tournament games to one database. On my PC it's called EngTourn.cbh, which is the default location. If you open this database, the whole match should be in there, with no missing games. (But be careful if you keep this db open in Chessbase while the match is running, as Fritz may not be able to write to it. It's best to not have it open in CB, but you can instead open EngTourn.cbh in Fritz prior to the match start, and Fritz will be able to add new games to it. That way you can see all the results while the match is running).

Note: if you run a match the default db for that is EngMatch.cbh

CL

[lkaufman]

amazing, & once again congratulations to you & don. it's great to have this pressure on robert (and you two!).

however, one flaw: unless i missed it, you are still running houdini with contempt. this is going to lead it to play, i believe, objectively suboptimal chess, even if it leads to better scores against inferior opponents. now that we seem to have an opponent on its level, its looking like contempt should be disabled for houdini.

Well, if so then it should also be disabled in Komodo. I think contempt is higher in Houdini, so this might slightly favor Houdini.

[yanquis1972]

i don't think i even realized komodo has contempt as a feature

however, given that i pretty much only use engines to analyze, i assume, as with houdini, komodo's comtempt is disabled in analyze mode. this can only be because the author(s) believe it to be the strongest possible setting.

i plan on re-running a tournament with komodo & houdini, & will disable contempt for both.

also plan on sitting back and watching the engines play each side of the yugoslav attack at tournament time control (will post the games if i do). i think i'll leave contempt on for that one.

[yanquis1972]

Fritz, as far as I know, saves all the tournament games to one database. On my PC it's called EngTourn.cbh, which is the default location. If you open this database, the whole match should be in there, with no missing games. (But be careful if you keep this db open in Chessbase while the match is running, as Fritz may not be able to write to it. It's best to not have it open in CB, but you can instead open EngTourn.cbh in Fritz prior to the match start, and Fritz will be able to add new games to it. That way you can see all the results while the match is running).

Note: if you run a match the default db for that is EngMatch.cbh

CL

yes, i created a new database (creatively titled 'new) for this match, & have edited the appropriate games to have the proper result. however, the tournament crashed (probably do to w/ this meddling) so i've continued, at least roughly where i left off, in yet another database. i'm assuming i can merge these databases when the tournament finishes.

[yanquis1972]

finally have the chance to observe some games. initial knee-jerk reactions, very much subject to change:

-komodo still has the classic feel in its ability to play what carl best called "elegant chess". posted example game in the main forum.

-houdini is simply better in endgames. more decisive evaluation, & better play. i'm not sure yet, but i don't think tablebases are a big factor in this.

updated results (the previous tournament crashed, & a couple openings have not been repeated from the given starting point for continuance, which is possibly to komodo's detriment as houdini went or a rampage after i restarted, & one win was in one of these games):

+14 =30 -18, with now 2 games that i need to let play out upon conclusion; one komodo is favored, another houdini more heavily favored.

[Don]

finally have the chance to observe some games. initial knee-jerk reactions, very much subject to change:

-komodo still has the classic feel in its ability to play what carl best called "elegant chess". posted example game in the main forum.

-houdini is simply better in endgames. more decisive evaluation, & better play. i'm not sure yet, but i don't think tablebases are a big factor in this.

I don't really know for sure, but that implies that Komodo is relatively better in the middlegame and I always believed it was the other way around.

There are some really basic endgames that we don't cover (that actually make no real difference in ELO in actual play) that may cause some to believe we are not as good in the endings but I have a feeling we are better there than you think.

There is a law of human nature too that says that we are too easily influenced by just one or two surprising events. In other words we are not very good judges. When I beat the top chess player in high school in a single game, I was suddenly consider the best player even though the reality was that he was better than me. Even when I lost game afterward that was the one that was remembered.

So I'm not really sure it's easy to say from a subjective standpoint. The time control algorithm can also change the ratio of how good you are in the opening vs the ending. Probably the only way to really instrument this is to play a bunch of matches from endgame positions. It could be that Houdini is just a little better in both or else a lot better in one but not the other.

updated results (the previous tournament crashed, & a couple openings have not been repeated from the given starting point for continuance, which is possibly to komodo's detriment as houdini went or a rampage after i restarted, & one win was in one of these games):

+14 =30 -18, with now 2 games that i need to let play out upon conclusion; one komodo is favored, another houdini more heavily favored.

[yanquis1972]

true, hence the heavy caveats. the few endgames i've observed, however, usually simply had a much more weighty evaluation than komodo, which implies to me that houdini is better at recognizing decisive endgames. i by no means meant to imply that komodo is poor in endgames, just that my impression thus far is that its inferior to houdini.

again coming from extremely limited observation, komodo handles quiet positions better than houdini, but i'm not so sure about more tactically complex or open ones. in the currently running set, houdini just defeated komodo with the white pieces of a fianchetto grunfeld after drawing with black. next up is an exchange grunfeld where i do not expect komodo to emerge with a win from either color, but we'll see.

there are tons of flaws, & bias, in these observations, of course, but i haven't run any engine matches in forever & the fact that houdini actually has a challenger has made me a bit giddy.

[Don]

true, hence the heavy caveats. the few endgames i've observed, however, usually simply had a much more weighty evaluation than komodo, which implies to me that houdini is better at recognizing decisive endgames. i by no means meant to imply that komodo is poor in endgames, just that my impression thus far is that its inferior to houdini.

And I cannot refute this either.

again coming from extremely limited observation, komodo handles quiet positions better than houdini, but i'm not so sure about more tactically complex or open ones. in the currently running set, houdini just defeated komodo with the white pieces of a fianchetto grunfeld after drawing with black. next up is an exchange grunfeld where i do not expect komodo to emerge with a win from either color, but we'll see.

there are tons of flaws, & bias, in these observations, of course, but i haven't run any engine matches in forever & the fact that houdini actually has a challenger has made me a bit giddy.

I know that we as humans like to label things, in order to wrap our brains around them. It gives us observational biases like crazy, but it seems to work too.