Komodo MP vs Houdini 3

[carldaman]

There are some really basic endgames that we don't cover (that actually make no real difference in ELO in actual play) that may cause some to believe we are not as good in the endings but I have a feeling we are better there than you think.

Hi Don, could you give an example of such a basic ending, or perhaps elaborate on what you mean by "not covered"? Komodo is a very well-rounded engine, not perfect, but very good at so many aspects of the game, that it would be jarring to see it fail in some basic endgame, regardless of whether it hurts the Elo or not. That would put off a lot of potential buyers - there are many people out there who tend to look for any minor flaw to talk themselves out of a mighty good engine (I saw people already complaining it has no TB support in another forum, while forgetting all the positive aspects of the engine).

Regards,
CL

[beram]

I am yet underway 100 games with my own testsuite
After 50 games with 4CPU 4m2s, I have a 41% score for Komodo 5.1MP
So this doesnt confirm your amazing results

Previous tests against Houdini3 with different TC's Komodo 5 scored 37-38 %

After 114 games (combining 4 and 12 core results as they are similar) Komodo MP is just one game behind. I will be the first to admit that this must be a lucky result for Komodo, while your 41% is probably unlucky. If the true result ends up somewhere in the middle, say 45% or so, this would still indicate better scaling for Komodo as on one core I would only expect around 40% against Houdini 3, as the one core version is only slightly stronger than Komodo 5 which you say scored 37-38%.

After 100 games I have this: 43% for Komodo 5.1

Code: Select all

Houdini 3 - Komodo 5.1, 4CPU, Blitz 4m+2s 
                       
1   Houdini 3 x64      +33/=48/-19 57.00%   57.0/100
2   Komodo 5.1 64-bit  +19/=48/-33 43.00%   43.0/100

[yanquis1972]

after a tremendous start, this looks to be roughly in line with what i expect from my 5+3 silver suite test. 86 of 100 games down, result +17 =43 -26, for 44.5%. this is very close to larry's 45% guesstimate if komodo can finish decently.

[ThatsIt]

We've 41.5% vs Houdini 3 x64 4CPU after 100 games too:
http://cegt.siteboard.eu/f6t824-testing ... -4cpu.html

Best wishes,
G.S.
(CEGT member)

[lkaufman]

I now have 158 games on 4 cores (3' + 2", 256mb hash, sse4 machines, Noomen testsuite) with Houdini 3 leading by 82.5 - 75.5. So about 48% for Komodo. This is a bit higher than others are reporting, which could be just sample error and/or a difference in hardware (for example if anyone is testing on non-sse4 machines we will do worse).
I got 49% in a hundred games the same way on 12 cores, which is better but not significantly so than 4 cores results. I'm doing the same 4 core vs 12 core with Critter 1.6a and so far the 12 core is doing much better, I'll report when done. So against Stockfish, Houdini, and Critter in all cases Komodo did better so far on 12 cores than on 4, though only marginally so vs. Houdini.

[yanquis1972]

from what kai posted in the main forum, doesn't it follow that komodo will be relatively stronger with more cores? i think your quad result is a bit lucky; i have seen a lot of reports in the low 40s & my 5+3 silver suite result was actually 46% (+22 =46 -30)(yes, its 98 games, not a hundred, i'm new to this), which was surprisingly high to me. i can certainly say, having watched quite a few of the games, that my impression has been that forced to choose between the two i would much rather have houdini at my disposal (hope you don't take offense to this; i am surprised i'm even saying it given expectations). seemed like a stabler evaluation, definitely sharper, & sniffs out results faster & with more emphasis. also, komodo almost always seemed to suffer to houdini in the late middlegame/endgame. an almost always inferior endgame analysis, incl. one shoddy example where a clearly drawn position was evaluated as advantage for one side (around +.80). rushed for the clippers & took a screencap because i figured you might be able to explain it quite simply: http://postimg.org/image/ebw3mlk21/

[lkaufman]

from what kai posted in the main forum, doesn't it follow that komodo will be relatively stronger with more cores? i think your quad result is a bit lucky; i have seen a lot of reports in the low 40s & my 5+3 silver suite result was actually 46% (+22 =46 -30)(yes, its 98 games, not a hundred, i'm new to this), which was surprisingly high to me. i can certainly say, having watched quite a few of the games, that my impression has been that forced to choose between the two i would much rather have houdini at my disposal (hope you don't take offense to this; i am surprised i'm even saying it given expectations). seemed like a stabler evaluation, definitely sharper, & sniffs out results faster & with more emphasis. also, komodo almost always seemed to suffer to houdini in the late middlegame/endgame. an almost always inferior endgame analysis, incl. one shoddy example where a clearly drawn position was evaluated as advantage for one side (around +.80). rushed for the clippers & took a screencap because i figured you might be able to explain it quite simply: http://postimg.org/image/ebw3mlk21/

Yes, my result seems a bit high at 48%. One detail is that I'm not using TBs in my test. From what I hear it's quite debatable whether they help at all (at least if on Hard Disk), but perhaps they do help Houdini a bit.
I agree that Houdini is probably a bit better in the latter part of the game on average; I think that Komodo is superior for opening analysis.
I don't think it follows that Komodo must be better with more cores; that depends on whether the elo gain at fixed depth continues to grow sufficiently with more cores. But so far my evidence is that it does benefit more from more cores than 4 than the other top three programs.
Regarding the endgame you post, it is rather arbitrary what score you give to rook and bishop vs. rook in general. I mean it is usually a theoretical draw but often won in practice. I think a score somewhere around a pawn for it is reasonable, but purists might like a lower score. In the given position the presence of a pawn for the weaker side does complicate things a bit.
In general this version does not have a lot of specific endgame knowledge, at least compared to other top programs. We have already added one basic endgame since the release, and will soon add others. In most cases TBs will make this irrelevant, and we will probably have TB support for next release.
Bear in mind that a tremendous amount of time went into getting the new program to work acceptably as MP, and a lot of other things have been neglected. This will no longer be an issue, so we expect good progress on all fronts now.

[carldaman]

In general this version does not have a lot of specific endgame knowledge, at least compared to other top programs. We have already added one basic endgame since the release, and will soon add others. In most cases TBs will make this irrelevant, and we will probably have TB support for next release.
Bear in mind that a tremendous amount of time went into getting the new program to work acceptably as MP, and a lot of other things have been neglected. This will no longer be an issue, so we expect good progress on all fronts now.

Hi Larry,
it's good that TB support will be added for the next release, as it's very important for analysis purposes, but for game play you should keep in mind many testers like to test without having TBs enabled. Therefore, I 'd hope you can add the basic endgame knowledge anyhow, rather than totally depending on the TB's. In my experience, it can be very deflating when an engine, especially a top commercial one, lacks important endgame knowledge.

(Example = King+ Rook Pawn + Wrong Bishop vs King, where the lone King will hold the draw if it can reach the corner in the front of the pawn, that the Bishop can't control. Many engines have no idea this frequent endgame is a clear draw, and instead misevaluate it as a win.)

Regards,
CL

[lkaufman]

In general this version does not have a lot of specific endgame knowledge, at least compared to other top programs. We have already added one basic endgame since the release, and will soon add others. In most cases TBs will make this irrelevant, and we will probably have TB support for next release.
Bear in mind that a tremendous amount of time went into getting the new program to work acceptably as MP, and a lot of other things have been neglected. This will no longer be an issue, so we expect good progress on all fronts now.

Hi Larry,
it's good that TB support will be added for the next release, as it's very important for analysis purposes, but for game play you should keep in mind many testers like to test without having TBs enabled. Therefore, I 'd hope you can add the basic endgame knowledge anyhow, rather than totally depending on the TB's. In my experience, it can be very deflating when an engine, especially a top commercial one, lacks important endgame knowledge.

(Example = King+ Rook Pawn + Wrong Bishop vs King, where the lone King will hold the draw if it can reach the corner in the front of the pawn, that the Bishop can't control. Many engines have no idea this frequent endgame is a clear draw, and instead misevaluate it as a win.)

Regards,
CL

Well, we do have that one, though not necessarily all the more complex versions of involving extra pawns. We just added the Philidor rook endgame draw right after the release, and will add others. We agree with you about not relying totally on TBs.

[beram]

I now have 158 games on 4 cores (3' + 2", 256mb hash, sse4 machines, Noomen testsuite) with Houdini 3 leading by 82.5 - 75.5. So about 48% for Komodo. This is a bit higher than others are reporting, which could be just sample error and/or a difference in hardware (for example if anyone is testing on non-sse4 machines we will do worse).
I got 49% in a hundred games the same way on 12 cores, which is better but not significantly so than 4 cores results. I'm doing the same 4 core vs 12 core with Critter 1.6a and so far the 12 core is doing much better, I'll report when done. So against Stockfish, Houdini, and Critter in all cases Komodo did better so far on 12 cores than on 4, though only marginally so vs. Houdini.

I am sorry to disappoint you but I now have this, 35% after 120 games 2x Noomen testsuite 2012:

Code: Select all

Houdini3 - Komodo5.1, Blitz 4m+2s 4CPU, AMD II X6 1090T
                       
1   Houdini 3 x64      +48/=60/-12 65.00%   78.0/120
2   Komodo 5.1 64-bit  +12/=60/-48 35.00%   42.0/120