Mythbuster continued

[Dr.Wael Deeb]

Don and Larry have been quiet: either they have not found this thread or have started MP version testing

Most probably they have serious problems with the MP version of Komodo
Dr.D

[beram]

Don and Larry have been quiet: either they have not found this thread or have started MP version testing

I think they are MP testing

Preliminary results by now:

Clash of the Titans by Timo Klaustermeyer Time controls 125min / 50 moves + (60min+5s) Ponder = ON
60 games Noomen testsuite 2011 30 pos Engines have 6 cores except in matches of Komodo4, 1 core
Fritz Benchmark for 1 thread: ~ 1900 kN/s

Code: Select all

Results after 388 of 480 games
HOUDINI:
Komodo4	Critter1.4	Stockfish2.2.2	Rybka 4.1	            total
25,5	        25,5	       30	            27,5	            108,5 points
48	          47	         48	            49	               192 games
53,1	        54,2	      62,5	          56,1                 56,5 %

KOMODO:
Houdini2.0C	Critter1.4	Stockfish2.2.2	Rybka 4.1	         total
22,5	        25,5	       25,5	             30	        103,5 points
48	          48	         49	               51	        196 games
46,9	        53,1	       52,0	             58,8	       52,8 %

[Houdini]

I think they are MP testing

No, Don is running his 6 sec/12 sec/24 sec/48 sec... test matches to prove scientifically that Komodo 4 overtakes Houdini at long time control.

Robert

[MM]

Don and Larry have been quiet: either they have not found this thread or have started MP version testing

Most probably they have serious problems with the MP version of Komodo
Dr.D

Hi Doc, I don't think so. I think they already have a relatively good MP version but i think they have serious problems to find some improvements that allow them to overtake (for real) Houdini and i don't think they will release anything untill they will reach this goal.

Best Regards

[TimoK]

I think they are MP testing

No, Don is running his 6 sec/12 sec/24 sec/48 sec... test matches to prove scientifically that Komodo 4 overtakes Houdini at long time control.

Robert

Ok, I know my test is statistically insufficiant, but after the games that have been played I'm strongly under the impression that Houdini has no scaling problem at all. So I wonder if Don will find any TC where Komodo 4 scores better than Houdini 2...

But ok, let's wait for Komodo MP, maybe Don and Larry have improved it a bit more so Houdini will have an opponent on an equal footing.

Best regards
Timo

[MM]

I think they are MP testing

No, Don is running his 6 sec/12 sec/24 sec/48 sec... test matches to prove scientifically that Komodo 4 overtakes Houdini at long time control.

Robert

Ok, I know my test is statistically insufficiant, but after the games that have been played I'm strongly under the impression that Houdini has no scaling problem at all. So I wonder if Don will find any TC where Komodo 4 scores better than Houdini 2...

But ok, let's wait for Komodo MP, maybe Don and Larry have improved it a bit more so Houdini will have an opponent on an equal footing.

Best regards
Timo

I think they have a priority: make a SP stronger than Houdini. After that they should try to build a very solid and stable MP. If this possibility would be true, Komodo MP would be still far.

Best Regards

[Dr.Wael Deeb]

Don and Larry have been quiet: either they have not found this thread or have started MP version testing

Most probably they have serious problems with the MP version of Komodo
Dr.D

Hi Doc, I don't think so. I think they already have a relatively good MP version but i think they have serious problems to find some improvements that allow them to overtake (for real) Houdini and i don't think they will release anything untill they will reach this goal.

Best Regards

Thanks And I hope you'll right
Dr.D

[beram]

Another game where Houdini lets Komodo come away with a draw from a lost position. After game 18. move 112. ..Kf8? This time it was in game 56
[D]7Q/8/3kpPp1/8/5n2/2P2PK1/1p6/1r6 b - - 0 49
after 49. ..Rg1 it is a draw, as Stockfish very quick and Komodo rather quickly see. Deep Rybka w32 on slow laptop after 6 minutes at depth 19 switches to Nh5.
Houdini 1.5a fails to find the winning move Nh5+! within reasonable time after depth 31 still it prefers Rg1 which is a draw as Komodo shows in his eval in the game.pgn just one move later after playing Kxf4

any explanations someone ?

[diep]

Another game where Houdini lets Komodo come away with a draw from a lost position. After This time it was in game 56
[D]7Q/8/3kpPp1/8/5n2/2P2PK1/1p6/1r6 b - - 0 49
after 49. ..Rg1 it is a draw, as Stockfish very quick and Komodo rather quickly see. Deep Rybka w32 on slow laptop after 6 minutes at depth 19 switches to Nh5.
Houdini 1.5a fails to find the winning move Nh5+! within reasonable time after depth 31 still it prefers Rg1 which is a draw as Komodo shows in his eval in the game.pgn just one move later after playing Kxf4

any explanations someone ?

Yeah the problem is the fact they just check a very narrow mainline, so if their first predicted move is some pawns difference, then huge problem occurs, as all pruning reduces the search tree a lot when material differences are there compared to the mainline.

So for example when you follow mainline and it gives +0.00 and some other line initially is +5 for white, then there is a material difference. The megapruning gets to effect and reduces all search lines bigtime, so checking Nh5 deeply isn't what happens, instead it prunes it so bigtime that it just can't find the win for it.

Similar effect is when first move is mispredicted. So it plays a move,
it expects move Y, instead move Z is best. Move Z it reduces heavily the search lines. Easily to less than half the depth it shows, and it just won't see much there.

All your examples show this clearly both for komodo as well as houdini, it's a common problem in todays computerchess.

In 1999 when i toyed with reductions myself, i couldn't get it stable for Diep; the bad side effects were worse than the benefit, as the nodes per second wasn't high yet of Diep. The world c hamps 1999 version of Diep used reductions; that's why in endgame it outsearched everyone so bigtime, even though it just got 70k nps or so at Bob's quad Xeon 400, versus others millions of nps.

In todays high nps game, this search depth difference is even bigger, just because they go easily through tactical barrier in any line, the reductions work and the worst case behaviour of them is more acceptable, even though it lets them miss weird things as you show.

Note that it's not just reductions in case of houdini + SF that cause this. The razoring + futility + bigger reduction factor in nullmove strengthens these effects.

Around 1995-1996 i was the first one to describe on RGCC the combination of using different R's together for nullmove. It's interesting to see how it seems to work over 10 years later for most of todays chess engines.

[beram]

Another game where Houdini lets Komodo come away with a draw from a lost position. After This time it was in game 56
[D]7Q/8/3kpPp1/8/5n2/2P2PK1/1p6/1r6 b - - 0 49
after 49. ..Rg1 it is a draw, as Stockfish very quick and Komodo rather quickly see. Deep Rybka w32 on slow laptop after 6 minutes at depth 19 switches to Nh5.
Houdini 1.5a fails to find the winning move Nh5+! within reasonable time after depth 31 still it prefers Rg1 which is a draw as Komodo shows in his eval in the game.pgn just one move later after playing Kxf4

any explanations someone ?

Yeah the problem is the fact they just check a very narrow mainline, so if their first predicted move is some pawns difference, then huge problem occurs, as all pruning reduces the search tree a lot when material differences are there compared to the mainline.

So for example when you follow mainline and it gives +0.00 and some other line initially is +5 for white, then there is a material difference. The megapruning gets to effect and reduces all search lines bigtime, so checking Nh5 deeply isn't what happens, instead it prunes it so bigtime that it just can't find the win for it.

Similar effect is when first move is mispredicted. So it plays a move,
it expects move Y, instead move Z is best. Move Z it reduces heavily the search lines. Easily to less than half the depth it shows, and it just won't see much there.

All your examples show this clearly both for komodo as well as houdini, it's a common problem in todays computerchess.

In 1999 when i toyed with reductions myself, i couldn't get it stable for Diep; the bad side effects were worse than the benefit, as the nodes per second wasn't high yet of Diep. The world c hamps 1999 version of Diep used reductions; that's why in endgame it outsearched everyone so bigtime, even though it just got 70k nps or so at Bob's quad Xeon 400, versus others millions of nps.

In todays high nps game, this search depth difference is even bigger, just because they go easily through tactical barrier in any line, the reductions work and the worst case behaviour of them is more acceptable, even though it lets them miss weird things as you show.

Note that it's not just reductions in case of houdini + SF that cause this. The razoring + futility + bigger reduction factor in nullmove strengthens these effects.

Around 1995-1996 i was the first one to describe on RGCC the combination of using different R's together for nullmove. It's interesting to see how it seems to work over 10 years later for most of todays chess engines.

thx for answering, but as Stockfish I read is also such a big pruner, why than he finds Nh5 so quick ? just good first move luck ?