Performances of engines in the Endgame

[Laskos]

I collected 360 hard 3-4-5 tablebase wins in 30-40 moves, and let engines without EGTB play against an enabled with 3-4-5 men Houdini from these positions.

Solved wins at 100ms/move:

Code: Select all

Komodo 5         66%
Stockfish 2.3.1  38%
Houdini 3        12%
Critter 1.6      11%
Rybka 4.1         7%

An amazing result for Komodo 5 at such short time control, and an abysmal Rybka 4.1 result.

Solved wins at 3,000ms/move

Code: Select all

Stockfish 2.3.1  80% (+42)
Komodo 5         70%  (+4)
Critter 1.6      53% (+42)
Houdini 3        46% (+34)
Rybka 4.1        39% (+32)

Komodo 5 improves by very little at x30 time control, and comes second to Stockfish. Komodo seems to have a very strong eval for endgames, Stockfish a good eval and search, so that at longer TC it surpasses Komodo in solving tablebase wins. Rybka is the engine which needs most tablebases.

[michiguel]

I collected 360 hard 3-4-5 tablebase wins in 30-40 moves, and let engines without EGTB play against an enabled with 3-4-5 men Houdini.

Solved wins at 100ms/move:

Code: Select all

Komodo 5         66%
Stockfish 2.3.1  38%
Houdini 3        12%
Critter 1.6      11%
Rybka 4.1         7%

An amazing result for Komodo 5 at such short time control, and an abysmal Rybka 4.1 result.

Solved wins at 3,000ms/move

Code: Select all

Stockfish 2.3.1  80% (+42)
Komodo 5         70%  (+4)
Critter 1.6      53% (+42)
Houdini 3        46% (+34)
Rybka 4.1        39% (+32)

Komodo 5 improves by very little at x30 time control, and comes second to Stockfish. Komodo seems to have a very strong eval for endgames, Stockfish a good eval and search, so that at longer TC it surpasses Komodo in solving tablebase wins. Rybka is the engine which needs most tablebases.

Could you send me the positions? I would like to try the latest Gaviota (w/o TBs, of course).

Miguel

[Laskos]

I collected 360 hard 3-4-5 tablebase wins in 30-40 moves, and let engines without EGTB play against an enabled with 3-4-5 men Houdini.

Solved wins at 100ms/move:

Code: Select all

Komodo 5         66%
Stockfish 2.3.1  38%
Houdini 3        12%
Critter 1.6      11%
Rybka 4.1         7%

An amazing result for Komodo 5 at such short time control, and an abysmal Rybka 4.1 result.

Solved wins at 3,000ms/move

Code: Select all

Stockfish 2.3.1  80% (+42)
Komodo 5         70%  (+4)
Critter 1.6      53% (+42)
Houdini 3        46% (+34)
Rybka 4.1        39% (+32)

Komodo 5 improves by very little at x30 time control, and comes second to Stockfish. Komodo seems to have a very strong eval for endgames, Stockfish a good eval and search, so that at longer TC it surpasses Komodo in solving tablebase wins. Rybka is the engine which needs most tablebases.

Could you send me the positions? I would like to try the latest Gaviota (w/o TBs, of course).

Miguel

I sent it to you via e-mail.

[gladius]

I collected 360 hard 3-4-5 tablebase wins in 30-40 moves, and let engines without EGTB play against an enabled with 3-4-5 men Houdini from these positions.

Very cool! Were the hard positions concentrated in any one area?

I'd love to see the positions to improve the Stockfish endgame eval. Thanks.

[Laskos]

I collected 360 hard 3-4-5 tablebase wins in 30-40 moves, and let engines without EGTB play against an enabled with 3-4-5 men Houdini from these positions.

Very cool! Were the hard positions concentrated in any one area?

I'd love to see the positions to improve the Stockfish endgame eval. Thanks.

I had an EPD file with some 15,000 TB wins, then using PGN-extract and EPD-utilities I found only 360 wins in 30-40 moves. Many are KRPKR or KBBKN, also the trivial KBNK. I uploaded it, see if it is of some use to you http://speedy.sh/ZJ4d8/360.epd

[michiguel]

I collected 360 hard 3-4-5 tablebase wins in 30-40 moves, and let engines without EGTB play against an enabled with 3-4-5 men Houdini.

Solved wins at 100ms/move:

Code: Select all

Komodo 5         66%
Stockfish 2.3.1  38%
Houdini 3        12%
Critter 1.6      11%
Rybka 4.1         7%

An amazing result for Komodo 5 at such short time control, and an abysmal Rybka 4.1 result.

Solved wins at 3,000ms/move

Code: Select all

Stockfish 2.3.1  80% (+42)
Komodo 5         70%  (+4)
Critter 1.6      53% (+42)
Houdini 3        46% (+34)
Rybka 4.1        39% (+32)

Komodo 5 improves by very little at x30 time control, and comes second to Stockfish. Komodo seems to have a very strong eval for endgames, Stockfish a good eval and search, so that at longer TC it surpasses Komodo in solving tablebase wins. Rybka is the engine which needs most tablebases.

Could you send me the positions? I would like to try the latest Gaviota (w/o TBs, of course).

Miguel

I sent it to you via e-mail.

Gaviota (noTB) at 40moves/4 sec (~100ms/move) in a 2.4 Ghz machine won 57% of the games against Gaviota+TBs.

Gaviota excelled at all positions that lead to KBNK, was poor at KRKN and KBBKN, and intermediate at the rest.

Miguel

[Laskos]

I collected 360 hard 3-4-5 tablebase wins in 30-40 moves, and let engines without EGTB play against an enabled with 3-4-5 men Houdini.

Solved wins at 100ms/move:

Code: Select all

Komodo 5         66%
Stockfish 2.3.1  38%
Houdini 3        12%
Critter 1.6      11%
Rybka 4.1         7%

An amazing result for Komodo 5 at such short time control, and an abysmal Rybka 4.1 result.

Solved wins at 3,000ms/move

Code: Select all

Stockfish 2.3.1  80% (+42)
Komodo 5         70%  (+4)
Critter 1.6      53% (+42)
Houdini 3        46% (+34)
Rybka 4.1        39% (+32)

Komodo 5 improves by very little at x30 time control, and comes second to Stockfish. Komodo seems to have a very strong eval for endgames, Stockfish a good eval and search, so that at longer TC it surpasses Komodo in solving tablebase wins. Rybka is the engine which needs most tablebases.

Could you send me the positions? I would like to try the latest Gaviota (w/o TBs, of course).

Miguel

I sent it to you via e-mail.

Gaviota (noTB) at 40moves/4 sec (~100ms/move) in a 2.4 Ghz machine won 57% of the games against Gaviota+TBs.

Gaviota excelled at all positions that lead to KBNK, was poor at KRKN and KBBKN, and intermediate at the rest.

Miguel

Wow! That's better than Houdini, Critter, Rybka at 3s/move on 3.6 GHz machine (on one core). That's extraordinary, you must have put a lot of endgame knowledge there. Curious how it improves at 40/120s.

[Laskos]

On these 360 hard 3-4-5 TB mates in 30-40 moves, without EGTB at 3,000ms/move the results are (number of wins on won positions of the engines without EGTB against Houdini 3 enabled with 3-4-5 Nalimovs)

Code: Select all

Stockfish 2.3.1  80% (+42) 
Komodo 5         70%  (+4)
Gaviota 0.86     66%  (+9) 
Hiarcs 14        64% (+26)
Hannibal 1.3     61%  (+8)
Shredder 12      58% (+20)
Critter 1.6      53% (+42) 
Houdini 3        46% (+34) 
Rybka 4.1        39% (+32) 
Junior 13        32% (+17)

The numbers in parenthesis are improvements from 100ms/move. Stockfish performs the best and improves the most on TB positions, Rybka 4.1 and Junior 13 are the worst in finding endgame TB wins.