AH_LTC Match: Komodo TCEC vs Houdini 4

[Aser Huerga]

AH_LTC Match: Komodo TCEC vs Houdini 4

Five i7-3930K CPUs 4.25 GHz
90'+30" TC
1 core for all engines
Ponder off
1024 Hash
3-4-5 EGTBs (when available) in SSDs
150 Early Starting Positions Suite, slightly tunned to avoid transpositions (checked with engine vs same-engine matches), and created as a proportional representation of the most played openings/variations on recent years in high quality human chess tournaments (source TWIC, only 2400+ ELO players) AH_150_Opening_Suite
All positions are played with Switched Colors for a total of 300 games per match
Games available for download, including eval/time/depth and PV for each move

Results:

Code: Select all

1 Komodo TCEC 3211 +77/-68/=155 51.50% 154.5/300
2 Houdini 4   3201 +68/-77/=155 48.50% 145.5/300

[/color]

Games: Komodo TCEC - Houdini 4


AH_LTC Stairway To Heaven competition

AH_LTC Made In Heaven class

Code: Select all

   # PLAYER                   : RATING  ERROR   POINTS  PLAYED    (%)
   1 Stockfish DD             : 3233.6   29.4    488.0     900   54.2%
   2 Komodo TCEC              : 3221.6   32.9    301.0     600   50.2%
     Stockfish_SZ 13110122    : 3210.8   23.6    486.0     900   54.0%
   3 Houdini 4                : 3207.3   27.5    592.5    1200   49.4%
     Komodo 6                 : 3180.8   23.0    564.0    1200   47.0%
     Houdini 3                : 3158.0   ----    268.5     600   44.8%
                                (Created by Ordo v0.8)

Code: Select all

1 Stockfish DD              :   900 (+232,=512,-156), 54.2 %

Komodo 6                      : 300 (+ 87,=176,- 37), 58.3 %
Houdini 4                     : 300 (+ 82,=155,- 63), 53.2 %
Komodo TCEC                   : 300 (+ 63,=181,- 56), 51.2 %

2 Komodo TCEC               :   600 (+133,=336,-131), 50.2 %

Houdini 4                     : 300 (+ 77,=155,- 68), 51.5 %
Stockfish DD                  : 300 (+ 56,=181,- 63), 48.8 %

  Stockfish_SZ 13110122     :   900 (+225,=522,-153), 54.0 %

Houdini 3                     : 300 (+ 94,=159,- 47), 57.8 %
Komodo 6                      : 300 (+ 70,=188,- 42), 54.7 %
Houdini 4                     : 300 (+ 61,=175,- 64), 49.5 %

3 Houdini 4                 :   1200 (+271,=643,-286), 49.4 %

Komodo 6                      : 300 (+ 76,=158,- 66), 51.7 %
Stockfish_SZ 13110122         : 300 (+ 64,=175,- 61), 50.5 %
Stockfish DD                  : 300 (+ 63,=155,- 82), 46.8 %
Komodo TCEC                   : 300 (+ 68,=155,- 77), 48.5 %

  Komodo 6                  :   1200 (+219,=690,-291), 47.0 %

Houdini 3                     : 300 (+ 74,=168,- 58), 52.7 %
Stockfish_SZ 13110122         : 300 (+ 42,=188,- 70), 45.3 %
Houdini 4                     : 300 (+ 66,=158,- 76), 48.3 %
Stockfish DD                  : 300 (+ 37,=176,- 87), 41.7 %

  Houdini 3                 :   600 (+105,=327,-168), 44.8 %

Komodo 6                      : 300 (+ 58,=168,- 74), 47.3 %
Stockfish_SZ 13110122         : 300 (+ 47,=159,- 94), 42.2 %

New Critter version (when available) will be the Candidate to enter in the Made In Heaven class.

Next Test: Made In Heaven class Time Control Comparison

[ouachita]

Larry should feel good about this result. On the other hand, Robert must feel as outnumbered as the British were by the main Zulu army.

[lkaufman]

Larry should feel good about this result. On the other hand, Robert must feel as outnumbered as the British were by the main Zulu army.

Yes, considering this result, the results of TCEC, and the results so far for the "3 Champs" tournament, in all of which Houdini 4 (or version just a couple weeks older in TCEC) finished clearly behind both Komodo and Stockfish, there isn't much doubt that Houdini has been relegated to third place for serious chess even if it is still top dog at blitz. Now it's a Komodo vs. Stockfish battle for supremacy. Has there been a single match report yet of any Houdini beating Komodo TCEC or any recent Stockfish version in a long match at 90' per side or more?

[Nelson Hernandez]

Do you mean at Rorke's Drift, Kambula, Gingindlovu, Eshowe or Ulundi?

[ouachita]

Has there been a single match report yet of any Houdini beating Komodo TCEC or any recent Stockfish version in a long match at 90' per side or more?

I am unaware of any test of >=60+ wherein H4 beat KTCEC or >=SF DD in a match or test. My review reveals that their ELO curves intersect around the 60+ mark, but mine is just an unscientific observation

[Vinvin]

Larry should feel good about this result. On the other hand, Robert must feel as outnumbered as the British were by the main Zulu army.

Yes, considering this result, the results of TCEC, and the results so far for the "3 Champs" tournament, in all of which Houdini 4 (or version just a couple weeks older in TCEC) finished clearly behind both Komodo and Stockfish, there isn't much doubt that Houdini has been relegated to third place for serious chess even if it is still top dog at blitz. Now it's a Komodo vs. Stockfish battle for supremacy. Has there been a single match report yet of any Houdini beating Komodo TCEC or any recent Stockfish version in a long match at 90' per side or more?

I think 90' per side still a bit short.
"3 champs" is 12 CPUs, 3 GHz, ponder=on, 90min+15sec. On 1 CPU that's similar to a time control around 10 to 15 min by move ponder=off ...

IMHO the time control where current H, K and SF are about equal is 60 sec by move on 4 cores 3 GHz ponder=off (or 180 sec by move on 1 core 3 GHz ponder=off).
Faster TC : Houdini becomes stronger , Slower TC : Komodo and SF become stronger.

[ouachita]

IMHO the time control where current H, K and SF are about equal is 60 sec by move on 4 cores 3 GHz ponder=off (or 180 sec by move on 1 core 3 GHz ponder=off).

I think we're saying the same thing, which is generally consistent with TCEC's results: H4 better at <60TC and SF and K better at >60TC.

[beram]

IMHO the time control where current H, K and SF are about equal is 60 sec by move on 4 cores 3 GHz ponder=off (or 180 sec by move on 1 core 3 GHz ponder=off).

I think we're saying the same thing, which is generally consistent with TCEC's results: H4 better at <60TC and SF and K better at >60TC.

No you are not saying the same and I agree with Vincent
So when average game lenght is agreed on about 80 moves than the TC barriers should be:
H4 better at < 120TC on 4 cores 3GHz ponder=off
and < 180TC on 1 core 3GHz ponder=off

happy testing

[lkaufman]

IMHO the time control where current H, K and SF are about equal is 60 sec by move on 4 cores 3 GHz ponder=off (or 180 sec by move on 1 core 3 GHz ponder=off).

I think we're saying the same thing, which is generally consistent with TCEC's results: H4 better at <60TC and SF and K better at >60TC.

No you are not saying the same and I agree with Vincent
So when average game lenght is agreed on about 80 moves than the TC barriers should be:
H4 better at < 120TC on 4 cores 3GHz ponder=off
and < 180TC on 1 core 3GHz ponder=off

happy testing

Well, the test which is the topic here was on one core and was far more games than the other tests, and at 90' + 30" Houdini 4 is relegated to third by a clear margin. So I'm inclined to think that on one core the breakeven level is around an hour for the game. It may not be shorter on 4 cores because I suspect that Houdini and SF have a better MP than Komodo for 4 cores but not for 12 or 16.

[beram]

When we check the results for CCRL 40/40 list (which is about 1 hour game), Houdini 4 is beating Komodo convincingly in personal match > 60% for Houdini
and the overall score is about 68-69% for Houdini 4 against 14 opponents, where Komodo TCEC scores 61-62% against the same. Although this can change of yet few games, I dont think that on this time control Komodo is already on par, far from that I would say.

Your argument about the 4 cores I agree upon