Timecontrol comparison - H4, SF, K-TCECr

Discussion of computer chess matches and engine tournaments.

Moderators: hgm, Rebel, chrisw

fastgm
Posts: 818
Joined: Mon Aug 19, 2013 6:57 pm

Timecontrol comparison - H4, SF, K-TCECr

Post by fastgm »

Code: Select all

Intel i5-3320M, 2.6 GHz
Default settings
EloStat Start Elo: 3000

15 sec. + 0.15 sec.  / Game

Games Completed = 2004 of 2004 (Avg game length = 48.706 sec)
Settings = RR/128MB/15000ms+150ms/M 800cp for 6 moves, D 350 moves/PGN:C:\8moves_v3.pgn(32000)

Time = 49265 sec elapsed, 0 sec remaining

1. Houdini 4 x64                587.0/1002  (+408,=358,-236)
2. Stockfish 020114 64 SSE4.2   493.5/1002  (+260,=467,-275)
3. Komodo TCECr 64-bit          477.0/1002  (+273,=408,-321)
4. Stockfish DD 64 SSE4.2       446.5/1002  (+208,=477,-317)


   Program                       Elo    +   -   Games   Score   Av.Op.  Draws
-------------------------------------------------------------------------------
1. Houdini 4 x64               : 3045   17  17  1002    58.6 %   2985   35.7 %
2. Stockfish 020114 64 SSE4.2  : 2996   16  16  1002    49.3 %   3001   46.6 %
3. Komodo TCECr 64-bit         : 2987   17  17  1002    47.6 %   3004   40.7 %
4. Stockfish DD 64 SSE4.2      : 2971   16  16  1002    44.6 %   3009   47.6 %


   White Wins   :    647 (32.3 %)
   Black Wins   :    502 (25.0 %)
   Draws        :    855 (42.7 %)

   White Perf.  : 53.6 %
   Black Perf.  : 46.4 %

----------------------------------------------------------------------------------------------


60 sec. + 0.60 sec. / Game

Games Completed = 2004 of 2004 (Avg game length = 197.142 sec)
Settings = RR/128MB/60000ms+600ms/M 800cp for 6 moves, D 350 moves/PGN:C:\8moves_v3.pgn(32000)

Time = 196984 sec elapsed, 0 sec remaining

1. Houdini 4 x64               549.0/1002  (+336,=426,-240)
2. Stockfish 020114 64 SSE4.2  521.5/1002  (+246,=551,-205)
3. Stockfish DD 64 SSE4.2      482.5/1002  (+216,=533,-253)
4. Komodo TCECr 64-bit         451.0/1002  (+224,=454,-324)


   Program                       Elo    +   -   Games   Score   Av.Op.  Draws
-------------------------------------------------------------------------------
1. Houdini 4 x64               : 3025   16  16  1002    54.8 %   2992   42.5 %
2. Stockfish 020114 64 SSE4.2  : 3011   14  14  1002    52.0 %   2996   55.0 %
3. Stockfish DD 64 SSE4.2      : 2990   15  15  1002    48.2 %   3003   53.2 %
4. Komodo TCECr 64-bit         : 2974   16  16  1002    45.0 %   3009   45.3 %


   White Wins   :    608 (30.3 %)
   Black Wins   :    414 (20.7 %)
   Draws        :    982 (49.0 %)

   White Perf.  : 54.8 %
   Black Perf.  : 45.2 %

------------------------------------------------------------------------------------------------

240 sec. + 2.40 sec. / Game

Games Completed = 2004 of 2004 (Avg game length = 779.616 sec)
Settings = RR/256MB/240000ms+2400ms/M 800cp for 6 moves, D 350 moves/PGN:C:\8moves_v3.pgn(32000)
Time = 778991 sec elapsed, 0 sec remaining

1. Stockfish 020114 64 SSE4.2   537.5/1002  (+262,=551,-189)
2. Houdini 4 x64                524.5/1002  (+288,=473,-241)
3. Stockfish DD 64 SSE4.2       487.5/1002  (+221,=533,-248)
4. Komodo TCECr 64-bit          454.5/1002  (+209,=491,-302)


   Program                       Elo    +   -   Games   Score   Av.Op.  Draws
-------------------------------------------------------------------------------
1. Stockfish 020114 64 SSE4.2  : 3019   14  14  1002    53.6 %   2994   55.0 %
2. Houdini 4 x64               : 3012   16  16  1002    52.3 %   2996   47.2 %
3. Stockfish DD 64 SSE4.2      : 2993   15  15  1002    48.7 %   3002   53.2 %
4. Komodo TCECr 64-bit         : 2976   15  15  1002    45.4 %   3008   49.0 %

                                     +-----------------------------------------------------------+
   White Wins   :    579 (28.9 %)    |                                15     60     240   15-240 |
   Black Wins   :    401 (20.0 %)    |  Program                       Elo    Elo    Elo    Diff  |
   Draws        :   1024 (51.1 %)    |-----------------------------------------------------------|
                                     |  Houdini 4 x64               : 3045   3025   3012   -33   |
   White Perf.  : 54.4 %             |  Stockfish 020114 64 SSE4.2  : 2996   3011   3019   +23   |
   Black Perf.  : 45.6 %             |  Stockfish DD 64 SSE4.2      : 2971   2990   2993   +22   |
                                     |  Komodo TCECr 64-bit         : 2987   2974   2976   -11   |
                                     +-----------------------------------------------------------+
Top
beram
Posts: 1187
Joined: Wed Jan 06, 2010 3:11 pm

Re: Timecontrol comparison - H4, SF, K-TCECr

Post by beram »

Thx Andreas,

Very nice comparing by doubling TC four times to see what ELO shifts arouse between those top engines
What I miss though is the specific result between the engines
For instance if engine A scores 50% against B at TC 1 what does it score at the other TC's
The results show already that SF performs better at longer time controls
But what does it give for Komodo TCEC in its matches against Houdini 4 ?
I expect to see something like 2-3 % going from TC 15 to TC 240.
According to Larry Kaufman, Komodo 6 would gain 8ELO(is approx 1%) for each time doubling against Houdini 4
Would be interesting to verify that
Top
fastgm
Posts: 818
Joined: Mon Aug 19, 2013 6:57 pm

Re: Timecontrol comparison - H4, SF, K-TCECr

Post by fastgm »

Hi Bram,

here are the specific result between the engines:

Code: Select all

Individual statistics:          15 sec. + 0.15 sec. 

1 Houdini 4 x64               : 3045  1002 (+408,=358,-236), 58.6 %

Stockfish 020114 64 SSE4.2    :        334 (+129,=121,- 84), 56.7 %
Stockfish DD 64 SSE4.2        :        334 (+136,=127,- 71), 59.7 %
Komodo TCECr 64-bit           :        334 (+143,=110,- 81), 59.3 %

2 Stockfish 020114 64 SSE4.2  : 2996  1002 (+260,=467,-275), 49.3 %

Houdini 4 x64                 :        334 (+ 84,=121,-129), 43.3 %
Stockfish DD 64 SSE4.2        :        334 (+ 75,=199,- 60), 52.2 %
Komodo TCECr 64-bit           :        334 (+101,=147,- 86), 52.2 %

3 Komodo TCECr 64-bit         : 2987  1002 (+273,=408,-321), 47.6 %

Houdini 4 x64                 :        334 (+ 81,=110,-143), 40.7 %
Stockfish 020114 64 SSE4.2    :        334 (+ 86,=147,-101), 47.8 %
Stockfish DD 64 SSE4.2        :        334 (+106,=151,- 77), 54.3 %

4 Stockfish DD 64 SSE4.2      : 2971  1002 (+208,=477,-317), 44.6 %

Houdini 4 x64                 :        334 (+ 71,=127,-136), 40.3 %
Stockfish 020114 64 SSE4.2    :        334 (+ 60,=199,- 75), 47.8 %
Komodo TCECr 64-bit           :        334 (+ 77,=151,-106), 45.7 %

-------------------------------------------------------------------

Individual statistics:          60 sec. + 0.60 sec.

1 Houdini 4 x64               : 3025  1002 (+336,=426,-240), 54.8 %

Stockfish 020114 64 SSE4.2    :        334 (+ 95,=152,- 87), 51.2 %
Stockfish DD 64 SSE4.2        :        334 (+116,=140,- 78), 55.7 %
Komodo TCECr 64-bit           :        334 (+125,=134,- 75), 57.5 %

2 Stockfish 020114 64 SSE4.2  : 3011  1002 (+246,=551,-205), 52.0 %

Houdini 4 x64                 :        334 (+ 87,=152,- 95), 48.8 %
Stockfish DD 64 SSE4.2        :        334 (+ 65,=236,- 33), 54.8 %
Komodo TCECr 64-bit           :        334 (+ 94,=163,- 77), 52.5 %

3 Stockfish DD 64 SSE4.2      : 2990  1002 (+216,=533,-253), 48.2 %

Houdini 4 x64                 :        334 (+ 78,=140,-116), 44.3 %
Stockfish 020114 64 SSE4.2    :        334 (+ 33,=236,- 65), 45.2 %
Komodo TCECr 64-bit           :        334 (+105,=157,- 72), 54.9 %

4 Komodo TCECr 64-bit         : 2974  1002 (+224,=454,-324), 45.0 %

Houdini 4 x64                 :        334 (+ 75,=134,-125), 42.5 %
Stockfish 020114 64 SSE4.2    :        334 (+ 77,=163,- 94), 47.5 %
Stockfish DD 64 SSE4.2        :        334 (+ 72,=157,-105), 45.1 %

-------------------------------------------------------------------

Individual statistics:          240 sec. + 2.40 sec.

1 Stockfish 020114 64 SSE4.2  : 3019  1002 (+262,=551,-189), 53.6 %

Houdini 4 Pro x64             :        334 (+100,=155,- 79), 53.1 %
Stockfish DD 64 SSE4.2        :        334 (+ 68,=223,- 43), 53.7 %
Komodo TCECr 64-bit           :        334 (+ 94,=173,- 67), 54.0 %

2 Houdini 4 Pro x64           : 3012  1002 (+288,=473,-241), 52.3 %

Stockfish DD 64 SSE4.2        :        334 (+105,=155,- 74), 54.6 %
Stockfish 020114 64 SSE4.2    :        334 (+ 79,=155,-100), 46.9 %
Komodo TCECr 64-bit           :        334 (+104,=163,- 67), 55.5 %

3 Stockfish DD 64 SSE4.2      : 2993  1002 (+221,=533,-248), 48.7 %

Houdini 4 Pro x64             :        334 (+ 74,=155,-105), 45.4 %
Stockfish 020114 64 SSE4.2    :        334 (+ 43,=223,- 68), 46.3 %
Komodo TCECr 64-bit           :        334 (+104,=155,- 75), 54.3 %

4 Komodo TCECr 64-bit         : 2976  1002 (+209,=491,-302), 45.4 %

Houdini 4 Pro x64             :        334 (+ 67,=163,-104), 44.5 %
Stockfish DD 64 SSE4.2        :        334 (+ 75,=155,-104), 45.7 %
Stockfish 020114 64 SSE4.2    :        334 (+ 67,=173,- 94), 46.0 %
Top
beram
Posts: 1187
Joined: Wed Jan 06, 2010 3:11 pm

Re: Timecontrol comparison - H4, SF, K-TCECr

Post by beram »

fastgm wrote:Hi Bram,

here are the specific result between the engines:

Code: Select all

Individual statistics:          15 sec. + 0.15 sec. 

1 Houdini 4 x64               : 3045  1002 (+408,=358,-236), 58.6 %

Stockfish 020114 64 SSE4.2    :        334 (+129,=121,- 84), 56.7 %
Stockfish DD 64 SSE4.2        :        334 (+136,=127,- 71), 59.7 %
Komodo TCECr 64-bit           :        334 (+143,=110,- 81), 59.3 %

2 Stockfish 020114 64 SSE4.2  : 2996  1002 (+260,=467,-275), 49.3 %

Houdini 4 x64                 :        334 (+ 84,=121,-129), 43.3 %
Stockfish DD 64 SSE4.2        :        334 (+ 75,=199,- 60), 52.2 %
Komodo TCECr 64-bit           :        334 (+101,=147,- 86), 52.2 %

3 Komodo TCECr 64-bit         : 2987  1002 (+273,=408,-321), 47.6 %

Houdini 4 x64                 :        334 (+ 81,=110,-143), 40.7 %
Stockfish 020114 64 SSE4.2    :        334 (+ 86,=147,-101), 47.8 %
Stockfish DD 64 SSE4.2        :        334 (+106,=151,- 77), 54.3 %

4 Stockfish DD 64 SSE4.2      : 2971  1002 (+208,=477,-317), 44.6 %

Houdini 4 x64                 :        334 (+ 71,=127,-136), 40.3 %
Stockfish 020114 64 SSE4.2    :        334 (+ 60,=199,- 75), 47.8 %
Komodo TCECr 64-bit           :        334 (+ 77,=151,-106), 45.7 %

-------------------------------------------------------------------

Individual statistics:          60 sec. + 0.60 sec.

1 Houdini 4 x64               : 3025  1002 (+336,=426,-240), 54.8 %

Stockfish 020114 64 SSE4.2    :        334 (+ 95,=152,- 87), 51.2 %
Stockfish DD 64 SSE4.2        :        334 (+116,=140,- 78), 55.7 %
Komodo TCECr 64-bit           :        334 (+125,=134,- 75), 57.5 %

2 Stockfish 020114 64 SSE4.2  : 3011  1002 (+246,=551,-205), 52.0 %

Houdini 4 x64                 :        334 (+ 87,=152,- 95), 48.8 %
Stockfish DD 64 SSE4.2        :        334 (+ 65,=236,- 33), 54.8 %
Komodo TCECr 64-bit           :        334 (+ 94,=163,- 77), 52.5 %

3 Stockfish DD 64 SSE4.2      : 2990  1002 (+216,=533,-253), 48.2 %

Houdini 4 x64                 :        334 (+ 78,=140,-116), 44.3 %
Stockfish 020114 64 SSE4.2    :        334 (+ 33,=236,- 65), 45.2 %
Komodo TCECr 64-bit           :        334 (+105,=157,- 72), 54.9 %

4 Komodo TCECr 64-bit         : 2974  1002 (+224,=454,-324), 45.0 %

Houdini 4 x64                 :        334 (+ 75,=134,-125), 42.5 %
Stockfish 020114 64 SSE4.2    :        334 (+ 77,=163,- 94), 47.5 %
Stockfish DD 64 SSE4.2        :        334 (+ 72,=157,-105), 45.1 %

-------------------------------------------------------------------

Individual statistics:          240 sec. + 2.40 sec.

1 Stockfish 020114 64 SSE4.2  : 3019  1002 (+262,=551,-189), 53.6 %

Houdini 4 Pro x64             :        334 (+100,=155,- 79), 53.1 %
Stockfish DD 64 SSE4.2        :        334 (+ 68,=223,- 43), 53.7 %
Komodo TCECr 64-bit           :        334 (+ 94,=173,- 67), 54.0 %

2 Houdini 4 Pro x64           : 3012  1002 (+288,=473,-241), 52.3 %

Stockfish DD 64 SSE4.2        :        334 (+105,=155,- 74), 54.6 %
Stockfish 020114 64 SSE4.2    :        334 (+ 79,=155,-100), 46.9 %
Komodo TCECr 64-bit           :        334 (+104,=163,- 67), 55.5 %

3 Stockfish DD 64 SSE4.2      : 2993  1002 (+221,=533,-248), 48.7 %

Houdini 4 Pro x64             :        334 (+ 74,=155,-105), 45.4 %
Stockfish 020114 64 SSE4.2    :        334 (+ 43,=223,- 68), 46.3 %
Komodo TCECr 64-bit           :        334 (+104,=155,- 75), 54.3 %

4 Komodo TCECr 64-bit         : 2976  1002 (+209,=491,-302), 45.4 %

Houdini 4 Pro x64             :        334 (+ 67,=163,-104), 44.5 %
Stockfish DD 64 SSE4.2        :        334 (+ 75,=155,-104), 45.7 %
Stockfish 020114 64 SSE4.2    :        334 (+ 67,=173,- 94), 46.0 %
thx Andreas

Well it seems that Larry could be right.
Komodo 6 TCEC goes from 40,7% to 42,5% to 44,5% gaining about approx 1% better performance (=7 ELO) with each time doubling going from 15 sec. + 0.15 sec. tot 240 sec. + 2.40 sec
If that trend would continue with each following time doubling, than K6 would get 49,5% at TC 64 min + 38,4 sec in your setup
It is a pity it would take to much time and effort to verify that

grts Bram
Top
beram
Posts: 1187
Joined: Wed Jan 06, 2010 3:11 pm

Re: Timecontrol comparison - H4, SF, K-TCECr

Post by beram »

Perhaps we can say something more about the trend of diminishing Elo gap between H and K by each time doubling
Andreas you have done another nice test between top 3 on TC 30 minutes (1000 games pro engine) to which alas until now no one responded
http://www.talkchess.com/forum/viewtopic.php?t=50860
In that test Komodo 6TCEC scored 46,9% against Houdini 4
If doubling theory would be true than here about 48% was to be expected

But lets not longer bother about number two and three anymore. A more important conclusion from that test was that Stockfish DD (already then) showed its superiority over Houdini and Komodo
From SF DD till version SF 270114 from now SF has gained another 30-40 ELO
So its hello and goodbye for Houdini and Komodo

Code: Select all

3 Komodo TCEC 64-bit        : 2986  1000 (+188,=564,-248), 47.0 % 

  Stockfish DD 64 SSE4.2    :        500 (+ 93,=285,-122), 47.1 % 
  Houdini 4 x64             :        500 (+ 95,=279,-126), 46.9 %
Top

Return to “Computer Chess Club: Tournaments and Matches”