Performances of engines in the Endgame

[Jouni]

If Scorpio means bitbases I am not surprised. In Rybka forum was examples how H3 was not able to mate with them!

[Laskos]

From openings:

Code: Select all

    Program                            Score       %     Elo    +   -

  1 Houdini 3 Pro x64              : 2075.0/2699  76.9   3166   16  15
  2 Stockfish 2.3.1 JA 64bit       : 1895.0/2784  68.1   3104   14  14
  3 Critter 1.6 64-bit             : 1281.0/2711  47.3   2985   13  13
  4 Komodo 5 64-bit                : 1158.0/2744  42.2   2961   13  13
  5 Deep Rybka 4.1 x64             :  551.0/2982  18.5   2802   16  16

Kai, I'm surprised by the result "from openings".
SF 2.3.1 appears to be significantly above Critter 1.6 in what probably are very fast games, that's not consistent with my experience.
Can you be more specific about the conditions, and can you also provide the full results including the draws?

Thanks,
Robert

It's simple, I used 50ms/move, while Houdini, Komodo and Critter obeyed this, Stockfish was using ~100ms, Rybka ~30ms. I was not interested in the absolute ratings, just difference from the second round-robin from endgame positions, at 50ms/move too, where the same happended. I am not using 2.5 + 0.05 because of the occasional time losses.

[Laskos]

If Scorpio means bitbases I am not surprised. In Rybka forum was examples how H3 was not able to mate with them!

Yes, Scorpio bitbases. I am not willing to download 7GB 3-4-5 Nalimov or Gaviota tablebases.

[Laskos]

Now I let them play with adjusted for strength time controls. Draws are kept this time.
Openings:

Code: Select all

    Program                            Score     %       Elo    +   -    Draws

  1 Junior 13                      : 2428.5/4620  52.6   3015    9   9   27.5 %
  2 Houdini 3                      : 2407.5/4630  52.0   3012    8   8   36.1 %
  3 Hiarcs 14                      : 2386.5/4629  51.6   3009    9   9   27.4 %
  4 Critter 1.6                    : 2339.0/4590  51.0   3006    8   8   33.3 %
  5 Komodo 5                       : 2348.5/4622  50.8   3005    8   8   31.4 %
  6 Rybka 4.1                      : 2152.0/4523  47.6   2986    8   8   33.1 %
  7 Stockfish 2.3.1                : 2048.0/4606  44.5   2967    8   8   31.8 % 

Endgames:

Code: Select all

    Program                            Score     %      Elo    +   -    Draws

  1 Komodo 5                       : 1271.5/2372  53.6   3022    9   9   58.5 %
  2 Hiarcs 14                      : 1265.0/2369  53.4   3020    9   9   54.1 %
  3 Stockfish 2.3.1                : 1210.5/2371  51.1   3006    9   9   56.1 %
  4 Rybka 4.1                      : 1171.5/2367  49.5   2997    9   9   58.4 %
  5 Junior 13                      : 1130.0/2368  47.7   2986    9  10   54.0 %
  6 Houdini 3                      : 1124.5/2373  47.4   2984    9   9   59.6 %
  7 Critter 1.6                    : 1120.0/2366  47.3   2984    9   9   57.9 % 

Endgame compared to general play, Elo points:

Code: Select all

Houdini 3      -28
Komodo 5       +17
Critter 1.6    -22
Stockfish 2.3  +39
Rybka 4.1      +11
Hiarcs 14      +11
Junior 13      -29

If taking into the account the error margins, it can be said with confidence that Stockfish overperfroms, Houdini 3 and Junior 13 underperform in the endgames.

[Houdini]

Thank you, fascinating.
Are your endgame positions available for download?

[Laskos]

Thank you, fascinating.
Are your endgame positions available for download?

http://speedy.sh/FaSnX/ENDGAME.epd

I have a larger one, but it needs to be polished, some positions have a forced outcome or contain too many heavy pieces.

[jdart]

This is not very surprising. Search in Stockfish is very selective and it reaches extreme depths in the endgame very quickly. Extra depth will probably help it there, compared to other engines, and probably helps more than the loss in precision from the selectivity hurts. (I don't much about the eval function of Stockfish, certainly not compared to other programs, so I can't say how much of a factor that is. I know it has recognizers for some common endgames, but that is pretty common nowadays).

--Jon

[Houdini]

Thank you, fascinating.
Are your endgame positions available for download?

http://speedy.sh/FaSnX/ENDGAME.epd

I have a larger one, but it needs to be polished, some positions have a forced outcome or contain too many heavy pieces.

Thank you, much appreciated - 884 positions is very useful already.

Cheers,
Robert

[Carlos Ylich]

Fantastic!
Great job. Congratulations !

[Adam Hair]

If Scorpio means bitbases I am not surprised. In Rybka forum was examples how H3 was not able to mate with them!

Yes, Scorpio bitbases. I am not willing to download 7GB 3-4-5 Nalimov or Gaviota tablebases.

You could generate your own 5 men Gaviota TBs, but you might not want to spend the time to do so. I think it took me two to three days to generate and compress them. I think it goes much faster on a better computer with more ram though.