which program is best in endgames

[George Tsavdaris]

I took 10 non-trivial endgame positions and played 4 engines with white and black (of course with tablebases). Result:

Code: Select all

                 
1   Stockfish 2.0 JA 64bit    12.5 - 7.5 10.0 - 10.0 11.5 - 8.5    34.0/60
2   Rybka 4                    7.5 - 12.5 11.5 - 8.5 11.5 - 8.5    30.5/60
3   Houdini 1.5a x64          10.0 - 10.0 8.5 - 11.5 9.5 - 10.5     28.0/60
4   Critter 0.90 64-bit       8.5 - 11.5 8.5 - 11.5 10.5 - 9.5    27.5/60

Can we have the games? Thanks.

[George Tsavdaris]

I performed a more extensive endgame testing at short time control with 50 initial positions (+reversed colours). Even if the same position was played several times, there are no two identical games, the randomness is very high. Each engine 1,600 games:

Code: Select all

    Program                            Score      %    Av.Op.  Elo    +   -    Draws

  1 Houdini 1.5a x64               : 945.5/1600  59.1   3190   3254   13  13   45.3 %
  2 Rybka 4_x64                    : 924.0/1600  57.8   3191   3246   12  12   51.2 %
  3 Ivanhoe B49jA                  : 893.5/1600  55.8   3193   3234   12  12   50.1 %
  4 Stockfish 2.0.1 JA 64bit       : 750.0/1600  46.9   3204   3182   12  12   49.1 %
  5 Komodo64 1.3 JA                : 727.0/1600  45.4   3206   3174   13  13   45.6 %
  6 Critter 0.90 64-bit            : 560.0/1600  35.0   3218   3111   13  13   43.2 %

Can we have the games? Thanks.

[Laskos]

I performed a more extensive endgame testing at short time control with 50 initial positions (+reversed colours). Even if the same position was played several times, there are no two identical games, the randomness is very high. Each engine 1,600 games:

Code: Select all

    Program                            Score      %    Av.Op.  Elo    +   -    Draws

  1 Houdini 1.5a x64               : 945.5/1600  59.1   3190   3254   13  13   45.3 %
  2 Rybka 4_x64                    : 924.0/1600  57.8   3191   3246   12  12   51.2 %
  3 Ivanhoe B49jA                  : 893.5/1600  55.8   3193   3234   12  12   50.1 %
  4 Stockfish 2.0.1 JA 64bit       : 750.0/1600  46.9   3204   3182   12  12   49.1 %
  5 Komodo64 1.3 JA                : 727.0/1600  45.4   3206   3174   13  13   45.6 %
  6 Critter 0.90 64-bit            : 560.0/1600  35.0   3218   3111   13  13   43.2 %

Can we have the games? Thanks.

Unfortunately I tested in LittleBlitzer which doesn't save games as PGN or something, only a large log file which I deleted.

[Laskos]

I played more games between Stockfish and Houdini and still Stockfish won, but of course result depends a lot from position selection. Critter is interesting: in some testsuites like EET it's simply the best, but not in real games!

Jouni

Indeed, Critter comes to the top 1-2-3 in almost every test suite, not only endgame. I will try to find another set of balanced non-trivial endgame positions, what is easier to find are test suites.

Kai

[slobo]

I wonder if somebody did endgame tests between top programs to test which program is best in that stage.

I know that houdini is better than stockfish or critter but it is not clear to me that houdini is best in all stages of the games and it is possible that a program is number 1 because it is better in the early stages of the game.

What kind of "best play in EG" do you mean?
With some sort of EGTB or without them?

Anyway, I don´t know the answer. I prefer not using EGTBs when testing engines.

[Jouni]

I did some tests with positions from EET suite. After 2*50 games Houdini wins "convincingly" against Stockfish 50,5-49,5. Houdini used Gaviota bases.

Jouni

[Kirk]

I wonder if somebody did endgame tests between top programs to test which program is best in that stage.

I know that houdini is better than stockfish or critter but it is not clear to me that houdini is best in all stages of the games and it is possible that a program is number 1 because it is better in the early stages of the game.

The "natural" ones often mentioned are KTULU and YACE

[PauloSoare]

I performed a more extensive endgame testing at short time control with 50 initial positions (+reversed colours). Even if the same position was played several times, there are no two identical games, the randomness is very high. Each engine 1,600 games:

Code: Select all

    Program                            Score      %    Av.Op.  Elo    +   -    Draws

  1 Houdini 1.5a x64               : 945.5/1600  59.1   3190   3254   13  13   45.3 %
  2 Rybka 4_x64                    : 924.0/1600  57.8   3191   3246   12  12   51.2 %
  3 Ivanhoe B49jA                  : 893.5/1600  55.8   3193   3234   12  12   50.1 %
  4 Stockfish 2.0.1 JA 64bit       : 750.0/1600  46.9   3204   3182   12  12   49.1 %
  5 Komodo64 1.3 JA                : 727.0/1600  45.4   3206   3174   13  13   45.6 %
  6 Critter 0.90 64-bit            : 560.0/1600  35.0   3218   3111   13  13   43.2 %

Houdini seems just a bit stronger than Rybka in the endgame, if at all. The gap between the first three and the rest is large, Stockfish seems not as good in the endgame as I suspected.

Was interesting to note that Houdini was 1-1.5 plies short of Rybka (adjusted +3 for depth) and IvanHoe, and 2-2.5 plies short of Stockfish, although its nps was the highest. Houdini seems to prune less in the endgame compared to some other engines. In the middlegame the search depth of Houdini is ~ equal to Rybka (+3) and IvanHoe. Contrary to what I felt, the main strength (those +60 Elo points) of Houdini seems to come more from the middlegame (although it's still strong in the endgame).

Kai

For analysis, if I have doubts in some positions, I prefer Stockfish.

[PauloSoare]

I wonder if somebody did endgame tests between top programs to test which program is best in that stage.

I know that houdini is better than stockfish or critter but it is not clear to me that houdini is best in all stages of the games and it is possible that a program is number 1 because it is better in the early stages of the game.

The "natural" ones often mentioned are KTULU and YACE

Maybe it was right 5 years ago.

[M ANSARI]

I am pretty sure that Rybka 4 is the best endgame engine, but at the moment it has a stall bug which makes analysis unreliable if you do end up with the bug hitting, so you would have to run the position several times while clearing hash. Houdini is also very good as it is based on R3 evaluation, and is much faster than R4 ... but it has some really poor changes in static evaluations than make it unreliable. In the end it is best to use several engines as this way you eliminate the blind spots that some engines (even R4 and H1.5) have. Hiarcs, Stockfish and even Crafty handle some endgames better than the top engines.