Houdini 3 vs ClusterRybka Game 4 ...

[Ajedrecista]

Hello!

Somehow the game was finished automaticaly but I think it doesn't matter:

[Event "105m+15s, unrated"]
[Site "Engine Room"]
[Date "2012.12.05"]
[Round "?"]
[White "Felix 2, Houdini 3 Pro x64"]
[Black "TryMe, Rybka Cluster 64 Cor"]
[Result "1-0"]
[PlyCount "135"]
[EventDate "2012.12.05"]
[TimeControl "6300+15"]

1. e4 c5 2. Nf3 Nc6 3. d4 cxd4 {0.11/21 97} 4. Nxd4 {0.23/28 182} e5 {0.17/22
198} 5. Nb5 {0.23/29 0} d6 {0.22/23 277} 6. N1c3 {0.28/30 0} Nf6 {0.22/23 332
(a6)} 7. Bg5 {0.32/29 134} a6 {0.26/21 0} 8. Na3 {0.45/30 478 (Lxf6)} Be6 {0.
27/22 305 (Le7)} 9. Nc4 {0.51/30 359 (Lxf6)} Rc8 {0.22/22 252} 10. Bxf6 {0.49/
30 0} Qxf6 {0.21/24 134} 11. Nb6 {0.49/31 0} Rb8 {0.23/24 85} 12. Ncd5 {0.48/
31 46 (Lc4)} Qg6 {0.21/24 267 (Dd8)} 13. Qd3 {0.70/28 153 (f3)} Be7 {0.08/22
117} 14. Nc7+ {0.70/29 15 (g3)} Kd8 {0.41/21 258 (Kf8)} 15. Ncd5 {0.72/28 147}
f5 {0.53/20 0} 16. Qc3 {0.70/25 286 (0-0-0)} fxe4 {0.84/21 428} 17. O-O-O {1.
01/27 0} Bxd5 {0.89/20 187 (Ke8)} 18. Nxd5 {0.91/26 53} Rc8 {0.84/19 67} 19.
Kb1 {0.94/28 269} Rf8 {1.03/20 0} 20. Qb3 {0.94/28 115} b5 {1.02/22 32} 21. Qa3
{0.90/28 209} Qg4 {1.00/22 0} 22. f3 {0.84/28 161} exf3 {0.99/22 0} 23. Nxe7 {
0.85/27 0} Kxe7 {1.00/22 84} 24. Qxd6+ {0.93/28 29} Kf7 {1.01/23 0} 25. gxf3 {
0.93/27 0} Qf5 {1.13/23 177} 26. Qd5+ {1.46/25 80 (Ld3)} Kf6 {0.80/22 138 (Ke8)
} 27. Bd3 {1.74/27 98} Qe6 {0.89/24 0} 28. Qe4 {1.90/27 119} g6 {1.22/21 0} 29.
h4 {2.09/27 391} Rcd8 {1.47/21 0} 30. a3 {1.84/28 118} Rd7 {1.35/21 14 (Td4)}
31. h5 {1.80/28 194} g5 {1.60/24 0} 32. h6 {1.86/25 340} Nd4 {1.96/22 99 (Tg8)}
33. Rhg1 {2.23/27 474 (Lf1)} Rfd8 {1.55/22 397} 34. Qe3 {2.35/30 0} Rg8 {1.51/
22 17} 35. Rde1 {2.30/28 66} Qd5 {1.63/22 46} 36. Rg4 {2.47/27 29} Qc5 {2.01/
22 206 (Dd6)} 37. Rg2 {2.43/26 59 (Teg1)} Qd5 {2.01/21 61} 38. Qg1 {2.43/24 0
(Le4)} Qf7 {1.60/21 241 (Dd6)} 39. Qf2 {2.51/25 59 (Dh2)} Rd5 {2.02/22 353
(Dd5)} 40. c3 {2.46/25 98 (Le4)} Ne6 {1.92/23 329} 41. Bc2 {2.53/28 0} Nf4 {2.
05/22 129} 42. Rg3 {2.60/26 21 (Db6+)} Rd7 {1.47/22 101 (Tgd8)} 43. Qe3 {2.69/
26 86} Qd5 {1.58/22 65} 44. Ka1 {2.71/26 7} Re8 {1.72/20 55 (Tc8)} 45. Rg4 {2.
28/25 68 (Dg1)} Ne6 {2.26/21 335 (Dc4)} 46. Bb1 {3.39/26 55 (Kb1)} Rc7 {2.62/
20 252 (Ted8)} 47. Ba2 {4.79/23 55} Qc5 {3.24/19 0} 48. Qe4 {5.39/26 60} Qd6 {
3.52/19 0} 49. Rg2 {5.63/27 95 (Teg1)} b4 {4.35/20 254} 50. axb4 {5.73/27 0}
Nf4 {4.46/19 22} 51. Bb1 {5.96/26 0} Qd7 {4.54/20 71} 52. Rd2 {6.34/27 0} Qa4+
{4.56/20 6} 53. Ba2 {6.68/26 27} Qc6 {4.68/20 0} 54. Qc2 {6.67/25 12} Rd7 {5.
48/21 273} 55. Rxd7 {6.96/28 43} Qxd7 {5.35/20 0} 56. Rd1 {6.42/27 29} Qc7 {5.
96/21 189} 57. Bb1 {6.47/26 0 (Da4)} e4 {5.05/21 5} 58. Qa4 {6.47/22 14} Qb8 {
4.84/19 62 (Te6)} 59. Qd7 {8.83/24 18 (Lxe4)} Re7 {5.50/15 4} 60. Rd6+ {9.26/
23 1} Ne6 {5.60/16 1} 61. Qc6 {9.53/22 1} Qb5 {6.37/15 3} 62. Qxe4 {9.52/22 0}
Qe5 {6.43/15 2} 63. Qd3 {10.06/21 0} Ra7 {8.08/19 33 (Db5)} 64. Rc6 {11.23/22
4 (Txa6)} a5 {5.79/16 5 (Tb7)} 65. b5 {11.86/22 5 (Dd8+)} Re7 {9.54/16 19 (Tb7)
} 66. b6 {19.45/22 32} a4 {14.16/19 98 (g4)} 67. Qd8 {25.90/24 29} Kf7 {27.50/
19 59} 68. Qc8 {29.93/24 0 TryMe,Rybka Cluster 64 Cor gibt auf (Lag: Av=0.42s,
max=1.2s)} 1-0



Bye
Ingo

Thank you very much for these games and also thanks for the Screenlap link! I followed more than half of the game.

I copied by hand the reached depths by each engine, so I can be wrong. This is the way I recorded them:

Code: Select all

(3rd move of Rybka) (4th move of Houdini)
(4th move of Rybka) (5th move of Houdini)

[...]

(67th move of Rybka) (68th move of Houdini)

Code: Select all

21 28
22 29
23 30
23 29
21 30
22 30
22 30
24 31
24 31
24 28
22 29
21 28
20 25
21 27
20 26
19 28
20 28
22 28
22 28
22 27
22 28
23 27
23 25
22 27
24 27
21 27
21 28
21 28
24 25
22 27
22 30
22 28
22 27
22 26
21 24
21 25
22 25
23 28
22 26
22 26
22 26
20 25
21 26
20 23
19 26
19 27
20 27
19 26
20 27
20 26
20 25
21 28
20 27
21 26
21 22
19 24
15 23
16 22
15 22
15 21
19 22
16 22
16 22
19 24
19 24

There are 65 moves without opening book by each engine. I recorded depths in this way because it was the easiest way for me when I looked to the PGN posted by Ingo in this thread.

Then I copied an own Fortran 95 code for calculating means and sample standard deviations and I did some changes. Here are the results I obtained:

Code: Select all

Number of moves without opening book by each engine: 65.
 
µ: average depth; s: sample standard deviation of depths.
 
µ_Rybka ~ 20.7538; s_Rybka ~ 2.1797
µ_Houdini ~ 26.4154; s_Houdini ~ 2.4102
 
µ_Houdini - µ_Rybka ~ 5.6615
sqrt[(s_Rybka)² + (s_Houdini)²] ~ 3.2496
k = (µ_Houdini - µ_Rybka)/sqrt[(s_Rybka)² + (s_Houdini)²] ~ 1.7422
 
(Roundings up to 0.0001).
Finished. Approximated elapsed time: 3042 µs.

IMHO, the higher is k, the better is for Houdini. So, averages depths of Rybka and Houdini are more less 20.75 and 26.41 for this game, respectively; of course, I did not take into account the time spent by each engine.

I hope that this info will be useful in some comparisons.

Regards from Spain.

Ajedrecista.

[M ANSARI]

With Rybka, you have to add 3 ply to get actual depth. This would still seem that Houdini 3 gets much deeper than Rybka ... with about maybe 2 or 3 ply.

[Ajedrecista]

Hello:

With Rybka, you have to add 3 ply to get actual depth. This would still seem that Houdini 3 gets much deeper than Rybka ... with about maybe 2 or 3 ply.

I already knew about this fact (I have always read three or four depths more, three depths more likely), as well as SF, Quazar and EXChess run very fast into depth.

This is why I gave the numbers as they are so people can reach their own conclusions. Rybka Cluster used 64 cores while Houdini 3 used 16 or 32 (I do not remember), so it seems that Rybka does not benefit in the same proportion of having too many cores, as Houdini does.

If we talk about real depths, then Houdini reaches in average at least two depths more than Rybka but using 50% (32 cores) or 25% (16 cores) of Rybka (64 cores). The numbers are telltale enough IMHO.

Regards from Spain.

Ajedrecista.

[beram]

Wow!!!

Wonderful game.

Some really good moves by Houdini. 16. Qc3!, 26. Qd5+! with the continuation 28.Qe4! and then 44. Ka1! with the intention of Bb1-Ba2.

Agree very good move Qc3 ! (perhaps 0-0-0 is just equally strong)
Houdini 3 on my AMD 6x 1080T doesn't find 16 Qc3 it stays with 16 f3 until I quit at depth 28
Can it have to do with hash table size ? I gave 2048 MB Hash

[Lavir]

Houdini 3 on my AMD 6x 1080T doesn't find 16 Qc3 it stays with 16 f3 until I quit at depth 28
Can it have to do with hash table size ? I gave 2048 MB Hash

Begin the analysis @ move 15 (Ncd5) till depth 27/28, then it will find it (usually it finds 0-0-0 or Qc3 as the continuation, it depends; both are the same thing).

When you analyze moves never begin from the same move you analyze, but a bit early, elsewhere the correct hash informations will not be stored (as it happens during a game).

[beram]

Houdini 3 on my AMD 6x 1080T doesn't find 16 Qc3 it stays with 16 f3 until I quit at depth 28
Can it have to do with hash table size ? I gave 2048 MB Hash

Begin the analysis @ move 15 (Ncd5) till depth 27/28, then it will find it (usually it finds 0-0-0 or Qc3 as the continuation, it depends; both are the same thing).

When you analyze moves never begin from the same move you analyze, but a bit early, elsewhere the correct hash informations will not be stored (as it happens during a game).

Thx, it worked but it took a long time >25min until depth 28 to find the continuation @ move 15 (Ncd5).
Komodo 3 and 5 @move 16 find it quick 1-2 minutes and Stockfish almost instantly.

I have also tried to give some meaningful commentary on the game

[Event "105m+15s, unrated"]
[Site "Engine Room"]
[Date "2012. 12. 05"]
[Round "?"]
[White "Felix 2, Houdini 3 Pro x64"]
[Black "TryMe, Rybka Cluster 64 Cor"]
[Result "*"]
[ECO "B33"]
[Annotator "Mourik,Bram + Houdini and Komodo"]


1. e4 c5 2. Nf3 Nc6 3. d4 cxd4 4. Nxd4 e5 5. Nb5 d6 6. N1c3 Nf6 7. Bg5 a6
8. Na3 Be6 9. Nc4 Rc8 10. Bxf6 Qxf6 11. Nb6 Rb8 12. Ncd5 Qg6 {This is
probably the first and already decisive mistake by black.. As we can see later
in the game, blacks king becomes too vulnerable. Theory gives and prefers Dd8
as mainline}13. Qd3 Be7 {Rybka evaluates pretty optimistic 0.08}
14. Nc7+ Kd8 15. Ncd5 f5 16. Qc3 ! { a very strong novelty } fxe4 17. O-O-O Bxd5 {what else,
black releases some pressure by exchanging one knight} 18. Nxd5 Rc8
19. Kb1 {white has al the time, the black knight cannot move from his position}Rf8 {there is no better move}
20. Qb3 {queen attacks pawn b7} b5 21. Qa3 {queen now attacks pawn a6} Qg4 22. f3 { a not so difficult tactical move, again black is forced to accept and
take the pawn}exf3 23. Nxe7 Kxe7 24. Qxd6+ { This was the idea, now
black king must go to f-file after which white retakes pawn on f3, black queen
cannot take back because of pinned position of queen and king on same
f-file} Kf7 25. gxf3 Qf5 26. Qd5+ Kf6 { one of the few black moves that aren't really forced, all the other moves where more or less forced.
Rybka judges this as 0,8 Houdini already 1.46 ! Fact still is, that black king cannot find a save haven}
27. Bd3 Qe6 28. Qe4 { of course white must not exchange pieces} g6
29. h4 {a logical and clear move, opening up the black kingside}Rcd8
30. a3 { a quiet but strong profylactic move, holding the black pawn on b5, taking one field for black knight, giving profylactic room for the king
and as seen later on, becomes a field for white bishop}Rd7 { there are no better options, all other moves will lead to same "boa constrictor" decisive pawn move h5!}
31. h5 {the decisive continuation} g5 32. h6 { now for sure, black king is jailed in his uncomfortably position in the centre}Nd4
33. Rhg1 Rfd8 34. Qe3 Rg8 35. Rde1 Qd5 36. Rg4 Qc5 37. Rg2 Qd5 38. Qg1 {looks mysterious but makes possible a change to h2 and tactical possibilities with pawnmove f4!?} Qf7
39. Qf2 Rd5 {Black king cannot move} 40. c3 {now black knight must leave the centre} Ne6
41. Bc2 Nf4 42. Rg3 { Houdini maintains the “boa style” Qb6 would leave to a more humanlike win
42. Qb6+ Qe6 43. Qxe6+ Nxe6 44. Bxh7 Re8 45. Be4 } 43. Qe3 Qd5 {how can white proceed ?}
44. Ka1{ by making room for bishop on a2} Re8 45. Rg4 Ne6
46. Bb1 {now we see the plan unfolding; exchanging bishop for knight and than attack on g5 and a6
combined} Rc7 47. Ba2 Qc5 48. Qe4 Qd6 49. Rg2 { there are quit a few other good moves but this one makes room for the white queen on g4}
b4 {in despair black offers a pawn, there are no better moves. But the white pressure and control over the position stays. We can stop commentary here} 1-0

[Lavir]

Thx, it worked but it took a long time >25min until depth 28 to find the continuation @ move 15 (Ncd5).
Komodo 3 and 5 @move 16 find it quick 1-2 minutes and Stockfish almost instantly.

The "velocity" with which it willl find it depends also on SMP randomness. Sometimes it is found as low as depth 25 (about 3-4 minutes) on move 15, other times it does take much more.

I think that contrary to popular believe Houdini 3 becomes stronger and stronger the more time (and so in the same way, the more hardware speed) it has, because it sorts of "builds up" move after move.

Good commentary, btw.

[M ANSARI]

Hello:

With Rybka, you have to add 3 ply to get actual depth. This would still seem that Houdini 3 gets much deeper than Rybka ... with about maybe 2 or 3 ply.

I already knew about this fact (I have always read three or four depths more, three depths more likely), as well as SF, Quazar and EXChess run very fast into depth.

This is why I gave the numbers as they are so people can reach their own conclusions. Rybka Cluster used 64 cores while Houdini 3 used 16 or 32 (I do not remember), so it seems that Rybka does not benefit in the same proportion of having too many cores, as Houdini does.

If we talk about real depths, then Houdini reaches in average at least two depths more than Rybka but using 50% (32 cores) or 25% (16 cores) of Rybka (64 cores). The numbers are telltale enough IMHO.

Regards from Spain.

Ajedrecista.

The core count might be correct but the way these cores are counted is completely different. Rybka 64 cluster was using cores on different motherboards and if it was using 4 motherboard at 16 cores each, it would be totally different than 1 motherboard using all 64 cores. So the hardware advantage of the Rybka Cluster is not what it seems like, it is more like a 16 core Rybka running MV at full depth for 3 best lines. So the Master 16 core motherboard would pick the best 3 lines and feed it to the other 3 slave motherboards and that would allow the best 3 lines to be searched at full power of the 16 cores. But really, Houdini 3 is so much more superior than Rybka 4.1 in search depth and search speed, plus some good improvement in evaluation, that Rybka Cluster would probably need 128 core cluster to be competitive with a 16 core H3. Either that or it would need a 64 core single motherboard shared memory setup to be competitive. It is obvious that H3 is simply out searching the 64 core cluster and totally outplaying it in every game. H3 is not like H1.5 or even H2.0 ... it is much much better evaluation wise and search wise, and this is showing at longer time controls. H3 is the new Rybka 3 of today and it will take more than throwing a bunch of cores at it with Rybka to be competitive. Rybka needs to be upgrade code wise as it is 3 years behind in that department.

What should be worrying about H3 to other engine developers is that just like Rybka 3 was, it still has some low lying ELO's to be gained by improving some evaluation values. One of these low lying fruit is that H3 still sees all rook endgames a pawn up as basically drawn and that gives some engines a lot of wins against it due to that fact. Since a large portion of endgames are rook endgames, fixing this one issue would remove a lot of losses for H3 and should push the ELO even higher. Maybe 7 piece EGTB's with rooks can be a shortcut to this as I am sure RH would have adjusted the evaluation accordingly if it was testing well.

[IWB]

..., it is more like a 16 core Rybka running MV at full depth for 3 best lines. So the Master 16 core motherboard would pick the best 3 lines and feed it to the other 3 slave motherboards and that would allow the best 3 lines to be searched at full power of the 16 cores....

THAT would be a very primitive way of running a cluster, ctually that bad that it doesn't really deserve the name "Cluster". In reality it should be more sophisticated like spliting the search tree into chunks distributing them and managing the results. ANd that is what Cluster Rybka is doing. I never had the feeling that it is doing something like optimistic pondering or other fancy stuff described in Gridchess/Cluster Toga for example.

I can only guess, but my "feeling" is that these 4x16 might be as effective as 1x32, or a little more. No prove, just gut feeling. I have seen 2 other cluster engines "in action" and think that cluster rybla is a pretty nice straight forward implementation of normal cluster theory without any bells and wistles ... and I know that I can be completly wrong

BYe
Ingo

[M ANSARI]

I am pretty sure that the way I described it is how the Rybka Cluster started. Of course it could have changed by now into a more sophisticated sharing of memory, but I have a feeling that it still works very much as initially described. Splitting at the root might work best for a certain setup, but you reach a point of diminished returns (especially on asymmetrical hardware or different motherboard hardware), where using the cores differently will get you much better results. My guess for that is around 64 cores or 128 cores. On a lot of positions you will get a "best move" much quicker if you use MPV with say 5 lines than you would with single PV. By much quicker I mean by a factor of 10X. Still ... H3 has quicker search and better evaluation, so chances are that it will be searching much deeper on the critical lines.