Mythbuster continued

[MM]

I think they are MP testing

No, Don is running his 6 sec/12 sec/24 sec/48 sec... test matches to prove scientifically that Komodo 4 overtakes Houdini at long time control.

Robert

Ok, I know my test is statistically insufficiant, but after the games that have been played I'm strongly under the impression that Houdini has no scaling problem at all. So I wonder if Don will find any TC where Komodo 4 scores better than Houdini 2...

But ok, let's wait for Komodo MP, maybe Don and Larry have improved it a bit more so Houdini will have an opponent on an equal footing.

Best regards
Timo

I think they have a priority: make a SP stronger than Houdini. After that they should try to build a very solid and stable MP. If this possibility would be true, Komodo MP would be still far.

Best Regards

[Dr.Wael Deeb]

Don and Larry have been quiet: either they have not found this thread or have started MP version testing

Most probably they have serious problems with the MP version of Komodo
Dr.D

Hi Doc, I don't think so. I think they already have a relatively good MP version but i think they have serious problems to find some improvements that allow them to overtake (for real) Houdini and i don't think they will release anything untill they will reach this goal.

Best Regards

Thanks And I hope you'll right
Dr.D

[beram]

Another game where Houdini lets Komodo come away with a draw from a lost position. After game 18. move 112. ..Kf8? This time it was in game 56
[D]7Q/8/3kpPp1/8/5n2/2P2PK1/1p6/1r6 b - - 0 49
after 49. ..Rg1 it is a draw, as Stockfish very quick and Komodo rather quickly see. Deep Rybka w32 on slow laptop after 6 minutes at depth 19 switches to Nh5.
Houdini 1.5a fails to find the winning move Nh5+! within reasonable time after depth 31 still it prefers Rg1 which is a draw as Komodo shows in his eval in the game.pgn just one move later after playing Kxf4

any explanations someone ?

[diep]

Another game where Houdini lets Komodo come away with a draw from a lost position. After This time it was in game 56
[D]7Q/8/3kpPp1/8/5n2/2P2PK1/1p6/1r6 b - - 0 49
after 49. ..Rg1 it is a draw, as Stockfish very quick and Komodo rather quickly see. Deep Rybka w32 on slow laptop after 6 minutes at depth 19 switches to Nh5.
Houdini 1.5a fails to find the winning move Nh5+! within reasonable time after depth 31 still it prefers Rg1 which is a draw as Komodo shows in his eval in the game.pgn just one move later after playing Kxf4

any explanations someone ?

Yeah the problem is the fact they just check a very narrow mainline, so if their first predicted move is some pawns difference, then huge problem occurs, as all pruning reduces the search tree a lot when material differences are there compared to the mainline.

So for example when you follow mainline and it gives +0.00 and some other line initially is +5 for white, then there is a material difference. The megapruning gets to effect and reduces all search lines bigtime, so checking Nh5 deeply isn't what happens, instead it prunes it so bigtime that it just can't find the win for it.

Similar effect is when first move is mispredicted. So it plays a move,
it expects move Y, instead move Z is best. Move Z it reduces heavily the search lines. Easily to less than half the depth it shows, and it just won't see much there.

All your examples show this clearly both for komodo as well as houdini, it's a common problem in todays computerchess.

In 1999 when i toyed with reductions myself, i couldn't get it stable for Diep; the bad side effects were worse than the benefit, as the nodes per second wasn't high yet of Diep. The world c hamps 1999 version of Diep used reductions; that's why in endgame it outsearched everyone so bigtime, even though it just got 70k nps or so at Bob's quad Xeon 400, versus others millions of nps.

In todays high nps game, this search depth difference is even bigger, just because they go easily through tactical barrier in any line, the reductions work and the worst case behaviour of them is more acceptable, even though it lets them miss weird things as you show.

Note that it's not just reductions in case of houdini + SF that cause this. The razoring + futility + bigger reduction factor in nullmove strengthens these effects.

Around 1995-1996 i was the first one to describe on RGCC the combination of using different R's together for nullmove. It's interesting to see how it seems to work over 10 years later for most of todays chess engines.

[beram]

Another game where Houdini lets Komodo come away with a draw from a lost position. After This time it was in game 56
[D]7Q/8/3kpPp1/8/5n2/2P2PK1/1p6/1r6 b - - 0 49
after 49. ..Rg1 it is a draw, as Stockfish very quick and Komodo rather quickly see. Deep Rybka w32 on slow laptop after 6 minutes at depth 19 switches to Nh5.
Houdini 1.5a fails to find the winning move Nh5+! within reasonable time after depth 31 still it prefers Rg1 which is a draw as Komodo shows in his eval in the game.pgn just one move later after playing Kxf4

any explanations someone ?

Yeah the problem is the fact they just check a very narrow mainline, so if their first predicted move is some pawns difference, then huge problem occurs, as all pruning reduces the search tree a lot when material differences are there compared to the mainline.

So for example when you follow mainline and it gives +0.00 and some other line initially is +5 for white, then there is a material difference. The megapruning gets to effect and reduces all search lines bigtime, so checking Nh5 deeply isn't what happens, instead it prunes it so bigtime that it just can't find the win for it.

Similar effect is when first move is mispredicted. So it plays a move,
it expects move Y, instead move Z is best. Move Z it reduces heavily the search lines. Easily to less than half the depth it shows, and it just won't see much there.

All your examples show this clearly both for komodo as well as houdini, it's a common problem in todays computerchess.

In 1999 when i toyed with reductions myself, i couldn't get it stable for Diep; the bad side effects were worse than the benefit, as the nodes per second wasn't high yet of Diep. The world c hamps 1999 version of Diep used reductions; that's why in endgame it outsearched everyone so bigtime, even though it just got 70k nps or so at Bob's quad Xeon 400, versus others millions of nps.

In todays high nps game, this search depth difference is even bigger, just because they go easily through tactical barrier in any line, the reductions work and the worst case behaviour of them is more acceptable, even though it lets them miss weird things as you show.

Note that it's not just reductions in case of houdini + SF that cause this. The razoring + futility + bigger reduction factor in nullmove strengthens these effects.

Around 1995-1996 i was the first one to describe on RGCC the combination of using different R's together for nullmove. It's interesting to see how it seems to work over 10 years later for most of todays chess engines.

thx for answering, but as Stockfish I read is also such a big pruner, why than he finds Nh5 so quick ? just good first move luck ?

[Houdini]

Another game where Houdini lets Komodo come away with a draw from a lost position. After game 18. move 112. ..Kf8? This time it was in game 56
[D]7Q/8/3kpPp1/8/5n2/2P2PK1/1p6/1r6 b - - 0 49
after 49. ..Rg1 it is a draw, as Stockfish very quick and Komodo rather quickly see. Deep Rybka w32 on slow laptop after 6 minutes at depth 19 switches to Nh5.
Houdini 1.5a fails to find the winning move Nh5+! within reasonable time after depth 31 still it prefers Rg1 which is a draw as Komodo shows in his eval in the game.pgn just one move later after playing Kxf4

any explanations someone ?

Interesting, it's another case where the "FiftyMoveDistance" option is useful. If you set it to 12, Houdini 2.0c finds Nh5! nearly immediately:

Code: Select all

 12/45	 0:00 	-3.66 	1...Rg1+ 2.Kxf4 g5+ 3.Ke3 b1Q 4.Qd8+ Ke5 5.Qd4+ Kf5 6.f7 Qc1+ 7.Qd2 Qa3 8.Kf2 Ra1 9.Kg3 Kf6 10.Qd7 Rg1+ 11.Kh3 (408.251) 3346
 13/45	 0:00 	-3.49--	1...Rg1+ 2.Kxf4 (549.357) 3662
 13/45	 0:00 	-3.14--	1...Rg1+ 2.Kxf4 (627.123) 3710
 13/45	 0:00 	-2.24--	1...Rg1+ 2.Kxf4 (858.314) 4146
 13/45	 0:00 	-2.59 	1...Nh5+ 2.Kf2 Rf1+ 3.Kxf1 b1Q+ 4.Kg2 Qb7 5.Kf2 e5 6.Qf8+ Ke6 7.Qe8+ Kxf6 8.Qf8+ Kg5 9.Qd8+ Kf5 10.Qd3+ e4 11.fxe4+ Qxe4 (2.012.941) 5121
 14/45	 0:00 	-2.46--	1...Nh5+ 2.Kf2 (2.701.993) 5361
 14/45	 0:00 	-2.54 	1...Nh5+ 2.Kf2 Rf1+ 3.Kxf1 b1Q+ 4.Kf2 Qb6+ 5.Kf1 Qa6+ 6.Ke1 Qa7 7.Kd2 e5 8.Kc2 Ke6 9.Qe8+ Kxf6 10.Qf8+ Qf7 11.Qd6+ Qe6 12.Qd8+ Qe7 13.Qb6+ Kf7 (3.899.927) 5611

Maybe I should look into using this parameter not just for analysis but also for game play...

Robert

[beram]

Another game where Houdini lets Komodo come away with a draw from a lost position. After game 18. move 112. ..Kf8? This time it was in game 56
[D]7Q/8/3kpPp1/8/5n2/2P2PK1/1p6/1r6 b - - 0 49
after 49. ..Rg1 it is a draw, as Stockfish very quick and Komodo rather quickly see. Deep Rybka w32 on slow laptop after 6 minutes at depth 19 switches to Nh5.
Houdini 1.5a fails to find the winning move Nh5+! within reasonable time after depth 31 still it prefers Rg1 which is a draw as Komodo shows in his eval in the game.pgn just one move later after playing Kxf4

any explanations someone ?

Interesting, it's another case where the "FiftyMoveDistance" option is useful. If you set it to 12, Houdini 2.0c finds Nh5! nearly immediately:

Code: Select all

 12/45	 0:00 	-3.66 	1...Rg1+ 2.Kxf4 g5+ 3.Ke3 b1Q 4.Qd8+ Ke5 5.Qd4+ Kf5 6.f7 Qc1+ 7.Qd2 Qa3 8.Kf2 Ra1 9.Kg3 Kf6 10.Qd7 Rg1+ 11.Kh3 (408.251) 3346
 13/45	 0:00 	-3.49--	1...Rg1+ 2.Kxf4 (549.357) 3662
 13/45	 0:00 	-3.14--	1...Rg1+ 2.Kxf4 (627.123) 3710
 13/45	 0:00 	-2.24--	1...Rg1+ 2.Kxf4 (858.314) 4146
 13/45	 0:00 	-2.59 	1...Nh5+ 2.Kf2 Rf1+ 3.Kxf1 b1Q+ 4.Kg2 Qb7 5.Kf2 e5 6.Qf8+ Ke6 7.Qe8+ Kxf6 8.Qf8+ Kg5 9.Qd8+ Kf5 10.Qd3+ e4 11.fxe4+ Qxe4 (2.012.941) 5121
 14/45	 0:00 	-2.46--	1...Nh5+ 2.Kf2 (2.701.993) 5361
 14/45	 0:00 	-2.54 	1...Nh5+ 2.Kf2 Rf1+ 3.Kxf1 b1Q+ 4.Kf2 Qb6+ 5.Kf1 Qa6+ 6.Ke1 Qa7 7.Kd2 e5 8.Kc2 Ke6 9.Qe8+ Kxf6 10.Qf8+ Qf7 11.Qd6+ Qe6 12.Qd8+ Qe7 13.Qb6+ Kf7 (3.899.927) 5611

Maybe I should look into using this parameter not just for analysis but also for game play...

Robert

[/size]
Thx, that explains it and what Vincent uttered, was not the case

[beram]

Conclusions:
The results of the matches in no way suggest a diminishing of performance(worse scaling) of Houdini at longer time control. Also they don’t point out that there is increasing performance(better scaling) of Komodo4 at longer time control.
The overall results in percentage are well in line with what you would expect them to be, when compared to results in their matches in CCRL, CEGT an IPON.
Also when going through the games played between them, you must conlude that Houdini is the best of the two.

Strong and weak points of Komodo and Houdini as been observed in their match Clash of the Titans.

Strong points Houdini:
Good in attacking the enemy kingside by opening it up with pawn sacrifice, so Houdini wins by kingside attack. See game 51 move f5! And game 47 move h5 !
Tactical sacrificing possibilities is sees faster. It sees them earlier than Komodo (or whatever other engine). Sacrificing pieces on the kingside opening up on the kingside by sacrificing a pawn etc. See game 38 with French opening C15, move 24. ..Bxa4! followed by 25. ..Bxb3!
Or game 59. The thematical Spanish sacrifice 29 Nf5!

Strong points Komodo:
In unbalanced material positions especially with pieces for the queen, Komodo evaluates better than Houdini. See game: 27 move 21..cxd4! where Houdini thinks he is better having the queen, while actually black should be better with his pieces in this relatively closed position. Also against Stockfish in game 60 as Vincent Lejeune also mentioned.
Good late middlegame/ early endgameplay - Examples are game 57 , Komodo nicely outplayed Houdini in a rook-knight endgame. And games 28 and 30 where won by very good endgame play with various pieces. Komodo sees the real advantage than much earlier.

Some other points:
Komodo: The fact of having the two bishops in many (semi-)closed positions is sometimes overrated by Komodo. It sometimes sees advantage of 0,7 0,8 where Houdini stucks on 0,2 0,3 and is able to prove right. See game 52

Houdini drawed two very nice games, it should have won. These games are already mentioned by me earlier. Game game 18. move 112. ..Kf8? and game 56 move 49. ..Rg1?
As I understood Robert Houdart well, these draws “could be fixed” by increasing the "FiftyMoveDistance factor” http://www.talkchess.com/forum/viewtopi ... 40&t=42802

Thx to Timo for organising this

grts Bram

[TimoK]

Thanks Bram for your great contributions (game analysis, discussions etc) to my tests! It was a great pleasure for me to read your interesting posts. I agree with the conclusions you drew.

All the best,
Timo