Komodo 3.0 first impressions!

[michiguel]

Yes, that's correct, but the results are somehow mixed. I put Komodo 3 against Houdini 1.5 at 2s + 0.2s, alternate colours:

The only problem is that Larry said that Komodo plays much worse against IPPOs in hyperfast games, far worse than in any other time control.

I took that into account, I just compared relative 2 results: from starting position 58% for Houdini, from that position 67%. I don't think there is an ambiguity here, Komodo evaluates better the position, Houdini misses it, but plays it better to the end.

Kai

Correct Kai, delta-delta-Elo is what matters in this case. However, there is an alternative explanation: Houdini plays badly from the opening position or K. plays very well from it. More positions may untie this.

Miguel

[M ANSARI]

If time were to reach the point of infinity, static evaluation would be by far the most important item in a chess engine. Unfortunately time is a major factor, and that is why search speed and good time control implementation is very important. A good time control algo that can effectively predict which move to play quickly can give the equivalent of a hardware advantage. The good news for Komodo though is that MP speed will quickly reach a point where adding more cores produces little speed increase. While using 2 cores gets a nice speed boost, this boost gets less and less with each core doubling. So IMHO the brute speed search of an engine will become less of an advantage as you reach 32 cores and higher, and thus MP Komodo might be able to close the speed search gap automatically.

Once increasing more and more cores will have less benefits, it will be the engine that uses these extra cores differently that will pull away from others.

[Laskos]

Yes, that's correct, but the results are somehow mixed. I put Komodo 3 against Houdini 1.5 at 2s + 0.2s, alternate colours:

The only problem is that Larry said that Komodo plays much worse against IPPOs in hyperfast games, far worse than in any other time control.

I took that into account, I just compared relative 2 results: from starting position 58% for Houdini, from that position 67%. I don't think there is an ambiguity here, Komodo evaluates better the position, Houdini misses it, but plays it better to the end.

Kai

Correct Kai, delta-delta-Elo is what matters in this case. However, there is an alternative explanation: Houdini plays badly from the opening position or K. plays very well from it. More positions may untie this.

Miguel

I did this a long time ago with Bob Hyatt's 4,000 EPD positions:

Code: Select all

    Program                     Score       %   Elo    +   -    Draws

  1 Houdini 1.5a            : 769.0/1344  57.2  3225   16  16   29.5 %
  2 Komodo64 3              : 575.0/1344  42.8  3175   16  16   29.5 %

I was just curious about the standard opening position, and it's pretty neutral in this case. My previous statements about the position in the present topic stand. 2s + 0.2s is not so hyper-fast, it's ~30s/game, if I put 0.5s + 0.05s, then the difference between Houdini and Komodo will seem much larger.

Kai

[lkaufman]

I think you can assume that our MP version will get a similar speedup on 4 or 6 cores as the other programs do, since the MP techniques are pretty well known. As for the tradeoff of speed for eval quality, it seems that the longer the time limit, the more important eval quality becomes. There is probably some time limit at which Komodo would score over 50% against Houdini, but it may be an impractical one like one day per move, so no way to prove that. But this is moot, because I think our latest private version would already score 50% at something like 40/2 hours, and our hope is that by our next release we can reach or exceed 50% at the slow tester time limits like 40/40' or 40/20'. For us to beat Houdini at blitz won't be so easy though due to our slow eval.

I am a huge fan of yours Larry since the 80's, I am glad you seem to want to take Houdini head on. This is a good thing, I did not get this feeling from Don. You can not ignore the top engine because you disagree with its politics. That will not wash with the customer IMO. I don't want to root for one program over another, but I can root for you.

Don is fully committed to passing Houdini. He doesn't think it is eligible for tournament play because most of the code is from the unknown author of Ippo, but he recognizes that the rating lists are more important than tournaments in practice so we cannot pretend Houdini doesn't exist.

[M ANSARI]

I think you can assume that our MP version will get a similar speedup on 4 or 6 cores as the other programs do, since the MP techniques are pretty well known. As for the tradeoff of speed for eval quality, it seems that the longer the time limit, the more important eval quality becomes. There is probably some time limit at which Komodo would score over 50% against Houdini, but it may be an impractical one like one day per move, so no way to prove that. But this is moot, because I think our latest private version would already score 50% at something like 40/2 hours, and our hope is that by our next release we can reach or exceed 50% at the slow tester time limits like 40/40' or 40/20'. For us to beat Houdini at blitz won't be so easy though due to our slow eval.

I am a huge fan of yours Larry since the 80's, I am glad you seem to want to take Houdini head on. This is a good thing, I did not get this feeling from Don. You can not ignore the top engine because you disagree with its politics. That will not wash with the customer IMO. I don't want to root for one program over another, but I can root for you.

Don is fully committed to passing Houdini. He doesn't think it is eligible for tournament play because most of the code is from the unknown author of Ippo, but he recognizes that the rating lists are more important than tournaments in practice so we cannot pretend Houdini doesn't exist.

Hi Larry. I think you have done a great job with the static evaluation of Komodo. I do hope that there will be a quick solution to improving the relative weakness in search speed. I guess by implementing MP that automatically improves speed. When that happens Komodo will be a real PIA for many engines as it simply sees things from a totally different perspective. As of yet I really don't feel the new static evaluation has been properly tested as Komodo will ruin many good positions simply due to missing simple tactical tricks. This is ironic as this is how many GM's would lose games against computers a few years back. Obviously computers are much stronger nowadays and a GM would be lucky to survive unscathed out of the opening ... but a couple of years ago it was always the GM outplaying the engine and then screwing up due to a tactical trick and sinking quickly.

[Jouni]

I don't think Komodo is weak in search speed / tactics! But may be it needs some more endgame knowledge. In match against Rybka often Rybka evaluates position as won, when Komodo is still at -1 or -2. Two examples:

[D]4B3/4n3/8/8/4K2P/2k3P1/4r3/8 w - - 0 56

Rybka -4,99 Komodo -1,44

[D]5r2/8/Pp6/3K2k1/4P3/6p1/8/R7 w - - 0 62

Rybka +4,98 Komodo +1,72

Jouni

[muxecoid]

Michael, that's honestly a nice position.
Did you set-up the position manually or was it taken from an actual game of Komodo 3.0? If it's the latter then it would be nice to see the full game, as this game looks interesting.

This position needs some explanation. I played a few games on FICS as black against stronger player who always played Halloween gambit. I lost all of them so I started looking for refutation using a computer and realized that in positions typical for early stages of this opening Komodo finds good moves faster.


1.e4 e5 2.Nf3 Nc6 3.Nc3 Nf6 4.Nxe5 Nxe5 5.d4 Nc6 6.d5 Ne5 7.f4 Ng6 8.e5
Ng8 9.d6 *

[D]r1bqkbnr/pppp1ppp/3P2n1/4P3/5P2/2N5/PPP3PP/R1BQKB1R b KQkq - 0 9

White is piece down but pawn at d6 looks dangerous. If white knight moves to b5 and than takes on c7 blach will lose the material advantage. Let's analyze the position with multi PV of two for 5 minutes (I have Q9300@ 2.7Ghz)...

( { [Komodo64 3] 22:-0.91} 9. ... cxd6 10.exd6 Qf6 11.Qe2+ Kd8 12.g3 Bxd6 13.Bd2 Qe7 14.O-O-O Qxe2 15.Bxe2 Nf6 16.Be3 Re8 17.Bd4 Bc7 18.Rhe1 Ne7 19.Bf3 Ba5 20.Be5 Nf5 21.Bxf6+ gxf6 22.Rxe8+ Kxe8 23.Nd5 )
( { [Komodo64 3] 21:+0.10} 9. ... Nh6 10.Qe2 Qh4+ 11.g3 Qg4 12.Nd5 Qxe2+ 13.Bxe2 cxd6 14.Nc7+ Kd8 15.Nxa8 b6 16.Nxb6 axb6 17.exd6 Bxd6 18.Be3 Kc7 19.O-O-O Nf5 20.Bf2 Bb7 21.Rhe1 Bc5 22.Bxc5 bxc5 23.Bg4 Nd6 24.f5 h5 25.Bh3 Nf8 26.Re5 c4 27.a4 )

And now Houdini 1.5 :

( { [Houdini 1.5a x64] 22:-0.83} 9.d6 cxd6 10.exd6 Qf6 11.Qe2+ Qe6 12.Nb5 Qxe2+ 13.Bxe2 Rb8 14.Be3 b6 15.O-O-O Nh4 16.g3 Nf5 17.Bf2 Nf6 18.Nxa7 Ne4 19.Bxb6 Rxb6 20.Nxc8 Rb8 21.Bd3 Nf2 22.Rde1+ Kd8 23.Bxf5 Nxh1 24.Ne7 Nf2 25.Re2 )
( { [Houdini 1.5a x64] 22:+0.18} 9.d6 Nh6 10.Qe2 Qh4+ 11.g3 Qg4 12.
Qxg4 Nxg4 13.Nd5 cxd6 14.Nc7+ Kd8 15.Nxa8 b6 16.Bg2 dxe5 17.h3 Nf6 18.fxe5 Nxe5 19.O-O Bc5+ 20.Kh2 Re8 21.Bg5 Be7 22.Bf4 Bc5 23.Rae1 d6 24.Bg5 Ba6 25.Bxf6+ gxf6 26.Rxf6 Ng4+ 27.hxg4 Rxe1 28.Rxf7 Re7 29.Rxe7 )

Looks similar, right? The catch here is that Nh6 was the second variation suggested by Komodo from the very beginning while Houdini preferred c6 during the first four minutes. (and if you re-run the test it may take even longer).

If c6 , known weak move is played Komodo immediately sees it is wrong, Houdini needs time to think.

This position is not typical and ability to play such position has low impact on overall strength but it shows the style of Komodo.
[/quote]

[lkaufman]

[quote="M ANSARIHi Larry. I think you have done a great job with the static evaluation of Komodo. I do hope that there will be a quick solution to improving the relative weakness in search speed. I guess by implementing MP that automatically improves speed. When that happens Komodo will be a real PIA for many engines as it simply sees things from a totally different perspective. As of yet I really don't feel the new static evaluation has been properly tested as Komodo will ruin many good positions simply due to missing simple tactical tricks. This is ironic as this is how many GM's would lose games against computers a few years back. Obviously computers are much stronger nowadays and a GM would be lucky to survive unscathed out of the opening ... but a couple of years ago it was always the GM outplaying the engine and then screwing up due to a tactical trick and sinking quickly.[/quote]

Could you clarify a couple things about your comments please?

1. "missing simple tactical tricks". Do you mean missing tactics that a human would see, or just missing some tactics that some other engine saw, but which are still difficult by any normal standard?
2. Are you comparing Komodo to MP programs, or are you saying that it is tactically weaker than other top SP engines? I know we are currently tactically weaker than Houdini SP and perhaps Critter SP, but I'm not so sure that we are weaker tactically than other SP engines.

In short, it's not clear to me that we have any tactical problem compared to other engines given the same number of nodes searched. If we do, we can try to address it, but if you are just saying that our NPS is relatively low we can only make small improvements in that without hurting the eval, aside from the huge speedup of MP of course.

[Eraserheads]

I read that Komodo is a very capable engine when it comes to producing positional novelties.

I would just like to share this exciting, wonderful game between Komodo 3 and my fav Gambit Tiger 2 (Aggressive).

The game started normally along the advance variation lines of the Caro Kann defense.

After GT's 16...Qa6, Komodo plays b4 offering a pawn. GT accepts, and Komodo manages to plant a pawn on b5, constricting Black's queenside.

[D]r3k2r/pp2nppp/q1n1p3/3pP3/1P1P3N/7P/P3NPP1/1R1Q1RK1 b kq -

I thought this was a good enough compensation for the sacked pawn, but what follows was an incredible plan by Komodo to mix things up on the Kingside, reminiscent of Kasparov's "momentum-powered" attacks. Kind of a 'crazy move' which one cannot easily see whether or not White can find enough compensation for the piece.

[D]rn2k2r/pp2nppp/4p3/1P1pP3/3P1N1N/7P/q4PP1/1R1Q1RK1 b kq -

Komodo offers a Knight, Black accepts, and the game went downhill very fast for Black after that.

Komodo throws the other Knight and after that Black's game is almost resignable.

[D]rn3r2/1p1knp1p/4N3/pP1pP3/3P3p/5Q1P/q4PP1/1RR3K1 b - -

Brilliant game by Komodo!





[Event "?"]
[Site "?"]
[Date "2011.09.25"]
[Round "?"]
[White "?"]
[Black "?"]
[Result "*"]

1. e4 {book 0s} c6 {3s (c5)} 2. d4 {book 0s} d5 {1s} 3. e5
{book 0s} Bf5 {4s} 4. Nf3 {book 0s} e6 {2s} 5. Be2 {book
0s} c5 {2s (Nd7)} 6. O-O {book 0s} Nc6 {3s} 7. c3 {book 0s}
cxd4 {2s} 8. cxd4 {book 0s} Nge7 {3s} 9. Nc3 {+0.24/13 3s}
Bg4 {5s (Ng6)} 10. Be3 {+0.15/13 6s} Nf5 {11s} 11. h3
{+0.22/15 8s} Bh5 {34s (Bxf3)} 12. Bg5 {+0.16/14 8s} Be7
{11s} 13. Bxe7 {+0.16/15 0s} Nfxe7 {3s} 14. Nh4 {+0.08/15
18s} Bxe2 {15s} 15. Nxe2 {+0.01/18 1s} Qb6 {3s} 16. Rb1
{+0.01/17 1s} Qa6 {17s (a5)} 17. b4 {+0.18/14 2s} Qxa2 {37s
(b5)} 18. b5 {+0.21/14 3s} Nb8 {18s} 19. Nf4 {+0.11/16 10s}
g5 {44s (Nd7)} 20. Nh5 {+0.50/15 14s} gxh4 {11s} 21. Ng7+
{+0.44/16 0s} Kd7 {21s (Kf8)} 22. Qf3 {+1.41/14 3s} Rf8
{6s} 23. Rfc1 {+0.97/17 2s} a5 {1s} 24. Nxe6 {+4.80/16 4s}
fxe6 {29s} 25. Qxf8 {+5.76/23 4s} Qxb1 {8s} 26. Rxb1
{+6.38/23 1s} b6 {54s} 27. Rc1 {+8.23/25 0s} a4 {11s}
28. Rc7+ {+9.20/20 5s} Kxc7 {3s} 29. Qxe7+ {+9.65/19 11s}
Nd7 {1s} 30. Qxe6 {+9.81/18 1s} Rc8 {2s} 31. Qd6+ {+9.97/18
1s} Kd8 {7s} 32. e6 {+10.78/19 0s} Rc7 {13s (Rc1+)} 33. e7+
{+11.04/18 2s} Ke8 {1s} 34. Qxc7 {+11.33/20 4s} Kxe7 {0s}
35. Qa7 {+11.49/19 1s} *

[Albert Silver]

I know we are currently tactically weaker than Houdini SP and perhaps Critter SP

I am not convinced either. As to tactics, I think Critter has a slight edge over Houdini, and I regard it as the current king of tactics.