Komodo3 test

[Tomcass]

For this I test have used my laptop, plus only one core per engine, so that the speed is not as high as usual.

Book HS-Mastersbook 7 moves
GUI Fritz 12
Time control 2 min plus 1 sec.
Ponder Off

201109Komodo3b-1 2011


1 Critter 1.2 64-bit SSE4(x1) 11.5 - 8.5 10.5 - 9.5 12.0 - 8.0** 34.0/60
2 Ivanhoe B47d x64x1 8.5 - 11.5 11.5 - 8.5 9.5 - 10.5 ** 29.5/60
3 Komodo64 3(x1) 9.5 - 10.5 8.5 - 11.510.5 - 9.5 ** 28.5/60
4 Deep Rybka 4 SSE42 x64x1 8.0 - 12.0 10.5 - 9.5 9.5 - 10.5 ** 28.0/60

Clear win for Critter 1.2. My feelings with Komodo3 have been very good. I like its playing style. I think that some extra speed is necessary to reach one of the top three places in the ranking.

The games in RTF and PGN.

megaupload.com 3UZJPDC0
megaupload.com H72KZG06

Regards

Tom

[lkaufman]

Our own tests confirm that Critter is especially strong (relative to other programs) at one minute plus one second on fast hardware, which is about the same as your time limit on a laptop. Apparently we catch up to Critter somewhere around twice this time limit. So there is no inconsistency between your result and the IPON list which puts Komodo comfortably ahead of Critter.

[Tomcass]

Thanks for your comment about my small test, Larry. As a reference, in my laptop average Kns per second are roughly

- Critter 1,800
- Ivanhoe 1,600
- Komodo 1,000
- Deep Rybka 100

This is about 12-13% only the speed they get in my usual computers.

I have just started a new tourney with exactly the same settings except Time Control. Instead of 2 min plus 1 sec this test is 5 min plus 1 sec. That is about double than the previous test. Let´s see the results in a couple of days.

Congratulations for the excellent work of your team to improve this program.

Tom.

[Tomcass]

For this I test have used my laptop, plus only one core per engine, so that the speed is not as high as usual.

Book HS-Mastersbook 7 moves
GUI Fritz 12
Time control 2 min plus 1 sec.
Ponder Off

201109Komodo3b-1 2011


1 Critter 1.2 64-bit SSE4(x1) 11.5 - 8.5 10.5 - 9.5 12.0 - 8.0** 34.0/60
2 Ivanhoe B47d x64x1 8.5 - 11.5 11.5 - 8.5 9.5 - 10.5 ** 29.5/60
3 Komodo64 3(x1) 9.5 - 10.5 8.5 - 11.510.5 - 9.5 ** 28.5/60
4 Deep Rybka 4 SSE42 x64x1 8.0 - 12.0 10.5 - 9.5 9.5 - 10.5 ** 28.0/60

Clear win for Critter 1.2. My feelings with Komodo3 have been very good. I like its playing style. I think that some extra speed is necessary to reach one of the top three places in the ranking.

Book HS-Mastersbook 7 moves
GUI Fritz 12
Time control 5 min plus 1 sec.
Ponder Off

201109Komodo3c 2011

1 Critter 1.2 64-bit SSE4(x1) 11.5 - 8.511.0 - 9.010.0 - 10.0** 32.5/60 0.00
2 Komodo64 3(x1) 8.5 - 11.512.0 - 8.012.0 - 8.0 ** 32.5/60 0.00
3 Ivanhoe B47d x64x1 9.0 - 11.08.0 - 12.013.0 - 7.0 ** 30.0/60
4 Deep Rybka 4 SSE42 x64x1 10.0 - 10.08.0 - 12.07.0 - 13.0 ** 25.0/60

The improvement of Komodo 3 when roughly doubling the Time Control seems to be clear, confirming Larry´s comment. The number of games, however, is still very small to conclude anything.

After these two short tournaments my feeling is that Komodo 3 is a truly different engine. I appreciate its style of playing and its ability to survive in difficult positions. Well done!.

Regards,

Tom.

The games http://www.megaupload.com/?d=80SNNFEZ

[lkaufman]

Thanks. I hope someone will play a match or tournament at a longer time control to see if this trend of improving results with greater depth continues. Who knows, maybe at 40/2 hours we might already be competitive with Houdini, but by the time anyone plays enough games to tell both programs will be outdated!

[PawnStormZ]

     Hi Larry.
 
       Thanks for all your work on Rybka 3 and now Komodo.  It is nice to see a GM using their knowledge and experience to make engines more "Human" in their evaluation and playing style.  I wish you continued success in your effort.
 
       After reading how Komodo 3 is so much improved, and getting good results in testing against even Houdini, I decided to run a long time-control match between Komodo 3 and Houdini 1.5.  The specifics and games can be found in the tourney forum.
 
       After the first 15 games or so the two were indeed close, with many draws.  I know that conclusions can not be made after so few games but I was kind of surprised as Houdini had jumped into a quick lead in the match that I did between it and Rybka 4.  However, Houdini has now taken 6 of the last 8 points to move ahead by 5 points after 25 games.  We will see what happens from here.

       There is definitely still much room for progress, even in the top engines.  In one game here Komodo had a lone King against K, P and WRONG B for Houdini.  Komodo did what it had to do, shuffling its King between a1 and b2 while for almost 30 moves Houdini refused to repeat.  Even though both had an eval close to 0 Houdini just kept "avoiding" the draw, which even its own eval must have seen.  I would like to see the engines just repeat and make the draw in positions where even weak players like myself know that it is a forced draw.

       Thanks again, and best of success to you and Don as you continue to improve Komodo!

                   Take care
 

[lkaufman]

Thanks for running the match. Regarding taking draws in dead drawn positions, note that the default setting for Komodo is to score a draw as -.05. We did this primarily so that Komodo will not take a quick draw as Black against a weaker engine just because it sees a chance to repeat. Since you are not playing a weaker opponent, but rather Houdini, the best results would be obtained by setting this value to zero. It probably wouldn't make a big difference, but if you play enough games there may be one in which Komodo needlessly avoids a repetition even though a tad worse otherwise. This would account for Komodo avoiding repetitions even with a bare king; this would not happen with the parameter set to zero.

[PawnStormZ]

     Hi Larry.
 
         I was not referring to Komodo as the "guilty party" in this case.  It only had the "shuffling" moves to avoid a loss, and WAS repeating.  It was Houdini that moved its B and K around forever, avoiding a repetition.  This happened in game 17.  Finally the evals got close enough to 0 that the GUI adjudicated it as a draw.
 
         Not sure why it is difficult (or is it that it takes away some thinking time) to have the engine "know" that K+B+P against K, when the B is wrong color, and K is already in the corner, is just a forced draw.  No decent engine would let its K leave the corner, so why does the opponent (Houdini in this case) keep moving around "waiting" for such a blunder?  Its own search must show that progress is not possible with good, not even best, play.
 
         Why not just "see" it is a forced draw, and repeat? Is it too costly speed-wise for the rest of the game to have this knowledge built in?
 
                           Take care

[lkaufman]

It's not a question of speed. I think that most programs evaluate endings such as KBP (wrong rook pawn) vs K as something like +.02 or so. The point is that the program should prefer to reach the good side of this ending to let's say just K vs. K, as the engine has no knowledge of whom the opponent may be; perhaps it's a beginner who doesn't know to keep the king in the corner. Also strong human players behave this way too; even if we know the result will be a draw anyway, we prefer to have the "sunny side" of the draw, if only to show the spectators which side was closer to winning. Now as for actually agreeing to the draw, if you want the engine to agree to a draw in such instances, you need to set contempt for a tiny negative value (maybe -0.03 or so), which in Komodo means setting draw score to a similar small positive value. This would however occasionally cause draws by repetitition in positions very slightly favorable to one side. If you don't round scores to the nearest centipawn (as most programs do) you could make this work almost perfectly, but that would entail a speed penalty. Maybe the best solution is to say that if the score stays below .05 or so for ten straight moves offer a draw.

[PawnStormZ]

Maybe the best solution is to say that if the score stays below .05 or so for ten straight moves offer a draw.

         Well, that is exactly the setting that I have in the GUI, which is why it was finally able to "stop the madness":!: