Komodo better at time controls like 1'+5" (Larry)

[Graham Banks]

Larry Kaufman posted the following:
http://rybkaforum.net/cgi-bin/rybkaforu ... ?tid=23945

No one seems to test using just increment, or primarily increment. This method has the advantage that it largely removes time allocation from the equation, as well as problems an engine might have with getting very short in time in long games. In short, it should correlate much better with how good the engine might be in analysis. We have some indications that Komodo does much better than normal against other engines at time controls like 1' + 5". If anyone would like to run such matches and post the result here it might prove interesting.

So Ray decided to run an experiment with this.

AMD Phenom II X6 @ 3.7Ghz
Time control 1' + 5"
ChessGUI
Ponder off
512MB hash
5-men Nalimov, Gaviota and Robbobases

Round-robin using Sherwin 50 test set, so 1500 games in total
All engines x64 and SSE where applicable, one core

FINAL results

Code: Select all

 1: Houdini 2.0c x64                 286.5 / 500 
 2: Komodo64 SSE Version 4           271.5 / 500 
 3: Deep Rybka 4.1 SSE42 x64         249.5 / 500 
 4: Critter 1.4 64-bit SSE4          242.0 / 500 
 5: Stockfish 2.2.1 JA 64bit SSE4.2  239.0 / 500 
 6: Ivanhoe 46h x64                  211.5 / 500 
-------------------------------------------------
1500 games: +475 =752 -273

BayesElo (mm 1 1)

Code: Select all

Rank Name                              Elo    +    - games score oppo. draws 
   1 Houdini 2.0c x64                   43   18   18   500   57%    -9   49% 
   2 Komodo64 SSE Version 4             27   18   18   500   54%    -5   47% 
   3 Deep Rybka 4.1 SSE42 x64           -2   17   17   500   50%     0   52% 
   4 Critter 1.4 64-bit SSE4           -10   18   18   500   48%     2   49% 
   5 Stockfish 2.2.1 JA 64bit SSE4.2   -14   18   18   500   48%     3   49% 
   6 Ivanhoe 46h x64                   -44   17   17   500   42%     9   54% 

[beram]

Hi Graham, nice you and Ray tested it

what did the resulttable look like?
how did Komodo4 score against Houdini2.0C and the other opponents?

grts bram

[lkaufman]

The result clearly confirms my hypothesis, regardless of specific match results. In no other blitz list are we this close to Houdini, and this far ahead of Critter 1.4.

[beram]

The result clearly confirms my hypothesis, regardless of specific match results. In no other blitz list are we this close to Houdini, and this far ahead of Critter 1.4.

You are right, so it could probably just take one other step forward with Komodo 4.1(?) to dethrone Houdini. Can you tell allready something more about the forthcoming release ?

grts Bram

[lkaufman]

Of course the big thing will be MP, which is almost but not quite ready for testing. Improved time management will be the next biggest change. Aside from that, some fairly modest search improvements or bug-fixes, depending on how you look at them. Hopefully it will add up to making Komodo a worthy rival for Houdini for the claim to being the world's best engine, especially at longer time controls.

[Graham Banks]

Hi Graham, nice you and Ray tested it

what did the resulttable look like?
how did Komodo4 score against Houdini2.0C and the other opponents?

grts bram

I'll ask Ray for the details and will post them when I get them.
You could probably work it out quickly from the pgn though - http://kirill-kryukov.com/chess/discuss ... p?id=25392

[beram]

Code: Select all

ComputerChess  2012
                            
1   Houdini 2.0c x64        +34/=46/-20 57.00%   57.0/100
2   Komodo64 SSE Version 4  +20/=46/-34 43.00%   43.0/100

[lkaufman]

This result is not favorable, but your result is only on 100 games whereas the very good results cited above were on 500 per version, so it does not appreciably alter the conclusion.