Can we have the games? Thanks.Jouni wrote:I took 10 non-trivial endgame positions and played 4 engines with white and black (of course with tablebases). Result:
Code: Select all
1 Stockfish 2.0 JA 64bit 12.5 - 7.5 10.0 - 10.0 11.5 - 8.5 34.0/60 2 Rybka 4 7.5 - 12.5 11.5 - 8.5 11.5 - 8.5 30.5/60 3 Houdini 1.5a x64 10.0 - 10.0 8.5 - 11.5 9.5 - 10.5 28.0/60 4 Critter 0.90 64-bit 8.5 - 11.5 8.5 - 11.5 10.5 - 9.5 27.5/60
which program is best in endgames
Moderators: hgm, Rebel, chrisw
-
- Posts: 1627
- Joined: Thu Mar 09, 2006 12:35 pm
Re: which program is best in endgames
After his son's birth they've asked him:
"Is it a boy or girl?"
YES! He replied.....
"Is it a boy or girl?"
YES! He replied.....
-
- Posts: 1627
- Joined: Thu Mar 09, 2006 12:35 pm
Re: which program is best in endgames
Can we have the games? Thanks.Laskos wrote:I performed a more extensive endgame testing at short time control with 50 initial positions (+reversed colours). Even if the same position was played several times, there are no two identical games, the randomness is very high. Each engine 1,600 games:
Code: Select all
Program Score % Av.Op. Elo + - Draws 1 Houdini 1.5a x64 : 945.5/1600 59.1 3190 3254 13 13 45.3 % 2 Rybka 4_x64 : 924.0/1600 57.8 3191 3246 12 12 51.2 % 3 Ivanhoe B49jA : 893.5/1600 55.8 3193 3234 12 12 50.1 % 4 Stockfish 2.0.1 JA 64bit : 750.0/1600 46.9 3204 3182 12 12 49.1 % 5 Komodo64 1.3 JA : 727.0/1600 45.4 3206 3174 13 13 45.6 % 6 Critter 0.90 64-bit : 560.0/1600 35.0 3218 3111 13 13 43.2 %
After his son's birth they've asked him:
"Is it a boy or girl?"
YES! He replied.....
"Is it a boy or girl?"
YES! He replied.....
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: which program is best in endgames
Unfortunately I tested in LittleBlitzer which doesn't save games as PGN or something, only a large log file which I deleted.George Tsavdaris wrote:Can we have the games? Thanks.Laskos wrote:I performed a more extensive endgame testing at short time control with 50 initial positions (+reversed colours). Even if the same position was played several times, there are no two identical games, the randomness is very high. Each engine 1,600 games:
Code: Select all
Program Score % Av.Op. Elo + - Draws 1 Houdini 1.5a x64 : 945.5/1600 59.1 3190 3254 13 13 45.3 % 2 Rybka 4_x64 : 924.0/1600 57.8 3191 3246 12 12 51.2 % 3 Ivanhoe B49jA : 893.5/1600 55.8 3193 3234 12 12 50.1 % 4 Stockfish 2.0.1 JA 64bit : 750.0/1600 46.9 3204 3182 12 12 49.1 % 5 Komodo64 1.3 JA : 727.0/1600 45.4 3206 3174 13 13 45.6 % 6 Critter 0.90 64-bit : 560.0/1600 35.0 3218 3111 13 13 43.2 %
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: which program is best in endgames
Indeed, Critter comes to the top 1-2-3 in almost every test suite, not only endgame. I will try to find another set of balanced non-trivial endgame positions, what is easier to find are test suites.Jouni wrote:I played more games between Stockfish and Houdini and still Stockfish won, but of course result depends a lot from position selection. Critter is interesting: in some testsuites like EET it's simply the best, but not in real games!
Jouni
Kai
-
- Posts: 2331
- Joined: Mon Apr 09, 2007 5:36 pm
Re: which program is best in endgames
What kind of "best play in EG" do you mean?Uri Blass wrote:I wonder if somebody did endgame tests between top programs to test which program is best in that stage.
I know that houdini is better than stockfish or critter but it is not clear to me that houdini is best in all stages of the games and it is possible that a program is number 1 because it is better in the early stages of the game.
With some sort of EGTB or without them?
Anyway, I don´t know the answer. I prefer not using EGTBs when testing engines.
"Well, I´m just a soul whose intentions are good,
Oh Lord, please don´t let me be misunderstood."
Oh Lord, please don´t let me be misunderstood."
-
- Posts: 3293
- Joined: Wed Mar 08, 2006 8:15 pm
Re: which program is best in endgames
I did some tests with positions from EET suite. After 2*50 games Houdini wins "convincingly" against Stockfish 50,5-49,5. Houdini used Gaviota bases.
Jouni
Jouni
-
- Posts: 5699
- Joined: Sat Mar 11, 2006 3:44 am
Re: which program is best in endgames
The "natural" ones often mentioned are KTULU and YACEUri Blass wrote:I wonder if somebody did endgame tests between top programs to test which program is best in that stage.
I know that houdini is better than stockfish or critter but it is not clear to me that houdini is best in all stages of the games and it is possible that a program is number 1 because it is better in the early stages of the game.
“He knew all the tricks, dramatic irony, metaphor, pathos, puns, parody, litotes and... satire. He was vicious”
-
- Posts: 1335
- Joined: Thu Mar 09, 2006 5:30 am
- Location: Cabo Frio, Brasil
Re: which program is best in endgames
For analysis, if I have doubts in some positions, I prefer Stockfish.Laskos wrote:I performed a more extensive endgame testing at short time control with 50 initial positions (+reversed colours). Even if the same position was played several times, there are no two identical games, the randomness is very high. Each engine 1,600 games:
Houdini seems just a bit stronger than Rybka in the endgame, if at all. The gap between the first three and the rest is large, Stockfish seems not as good in the endgame as I suspected.Code: Select all
Program Score % Av.Op. Elo + - Draws 1 Houdini 1.5a x64 : 945.5/1600 59.1 3190 3254 13 13 45.3 % 2 Rybka 4_x64 : 924.0/1600 57.8 3191 3246 12 12 51.2 % 3 Ivanhoe B49jA : 893.5/1600 55.8 3193 3234 12 12 50.1 % 4 Stockfish 2.0.1 JA 64bit : 750.0/1600 46.9 3204 3182 12 12 49.1 % 5 Komodo64 1.3 JA : 727.0/1600 45.4 3206 3174 13 13 45.6 % 6 Critter 0.90 64-bit : 560.0/1600 35.0 3218 3111 13 13 43.2 %
Was interesting to note that Houdini was 1-1.5 plies short of Rybka (adjusted +3 for depth) and IvanHoe, and 2-2.5 plies short of Stockfish, although its nps was the highest. Houdini seems to prune less in the endgame compared to some other engines. In the middlegame the search depth of Houdini is ~ equal to Rybka (+3) and IvanHoe. Contrary to what I felt, the main strength (those +60 Elo points) of Houdini seems to come more from the middlegame (although it's still strong in the endgame).
Kai
-
- Posts: 1335
- Joined: Thu Mar 09, 2006 5:30 am
- Location: Cabo Frio, Brasil
Re: which program is best in endgames
Maybe it was right 5 years ago.Kirk wrote:The "natural" ones often mentioned are KTULU and YACEUri Blass wrote:I wonder if somebody did endgame tests between top programs to test which program is best in that stage.
I know that houdini is better than stockfish or critter but it is not clear to me that houdini is best in all stages of the games and it is possible that a program is number 1 because it is better in the early stages of the game.
-
- Posts: 3707
- Joined: Thu Mar 16, 2006 7:10 pm
Re: which program is best in endgames
I am pretty sure that Rybka 4 is the best endgame engine, but at the moment it has a stall bug which makes analysis unreliable if you do end up with the bug hitting, so you would have to run the position several times while clearing hash. Houdini is also very good as it is based on R3 evaluation, and is much faster than R4 ... but it has some really poor changes in static evaluations than make it unreliable. In the end it is best to use several engines as this way you eliminate the blind spots that some engines (even R4 and H1.5) have. Hiarcs, Stockfish and even Crafty handle some endgames better than the top engines.