Deep Rybka 4.1 vs Houdini 3

[MM]

I think we may need many games to have a clear result but i think that Rybka 4.1 is still, perhaps except Komodo, the best engine for ''understanding'' positions.

So i can understand that Houdini generally can have some trouble facing Rybka at long time control, especially with many cores.

What would surprise me is if Houdini 3 should show a worse result compared to Houdini 1.5a playing Rybka 4.1.

I agree that the strenght of an engine must be calculated on many individual matches but i wouldn't understand if the improvement of Houdini 3 caused (if) a drop in the performance against Rybka 4.1.



Best Regards

In a rating list gauntlet inevitably some individual matches go well, other go poorly.

Against "Deep Rybka 4.1 x64 4CPU" Houdini 3 had a poor run and performed 50 Elo below its rating.
But against "Deep Rybka 4 Exp.61 x64 1CPU" Houdini 3 had a good run and performed 45 Elo above its rating.

It is not smart to attach a lot of importance to these individual results. There is a reason why 1000 games are played for a rating list, not 50.

Robert

I'm not talking about ''general performance''. I'm just stating that it is strange that the newer and the stronger version of the same engine (Houdini 3) performs worse than the weaker and the oldest version of the same engine (Houdini 1.5a) against a common opponent, Rybka 4.1 4cpu.

Theoretically if there's a difference around 70 elo between 2 versions of the same engine (H3 and H1.5a) the strongest version should have a better performance against a common opponent.


Said that, Houdini 3 remains the unmatched leader of the chess engines as all charts show, at least at short and medium/long time control.

Best Regards

[Houdini]

Theoretically if there's a difference around 70 elo between 2 versions of the same engine (H3 and H1.5a) the strongest version should have a better performance against a common opponent.

Only if the number of games is sufficient!

Rybka 4 is rated 3048, has a performance of 3110 ± 70 against Houdini 3 and of 2997 ± 50 against Houdini 1.5a.
Everything falls nicely within the error margins of the gauntlets.

You've picked probably the only opponent against which Houdini 1.5a scored better than Houdini 3, and then discuss this result as if it was not a statistical outlier.

Robert

P.S. It always appears to be difficult to discuss these elementary statistics on computer chess forums... it's rather frustrating for me.

[lech]

[quote="Houdini"
P.S. It always appears to be difficult to discuss these elementary statistics on computer chess forums... it's rather frustrating for me.[/quote]
If I read your avatar well, your left hand play chess, and the right is ready to beat.
Use the right hand.

[MM]

Theoretically if there's a difference around 70 elo between 2 versions of the same engine (H3 and H1.5a) the strongest version should have a better performance against a common opponent.

Only if the number of games is sufficient!

Rybka 4 is rated 3048, has a performance of 3110 ± 70 against Houdini 3 and of 2997 ± 50 against Houdini 1.5a.
Everything falls nicely within the error margins of the gauntlets.

You've picked probably the only opponent against which Houdini 1.5a scored better than Houdini 3, and then discuss this result as if it was not a statistical outlier.

Robert

P.S. It always appears to be difficult to discuss these elementary statistics on computer chess forums... it's rather frustrating for me.

What i stated is clear and published by CEGT. Even not so many games have been played the result is anyway surprising, that' s pretty obvious.

You are trying to demonstrate what i didn't say or perhaps i didn't explain well my thoughts.

I stated that not so many games were played but the result is surprising anyway.

I expected you to say something more than talking about the error bar.

I remember you ran matches at long time control against Stockfish, Komodo and H2 but not Rybka 4.1. I remember that you talked about a good algorithm for long time control in H3.

If you really believe that is just a question of number of games and error bar if was you i would set up 2 matches at long time control H1.5a-rybka 4.1 and H3-rybka 4.1 4 cores each with 500 games each or more. You have the machine to do it. Me i haven't.
That would be very interesting.

Best Regards

[tomgdrums]

Theoretically if there's a difference around 70 elo between 2 versions of the same engine (H3 and H1.5a) the strongest version should have a better performance against a common opponent.

Only if the number of games is sufficient!

Rybka 4 is rated 3048, has a performance of 3110 ± 70 against Houdini 3 and of 2997 ± 50 against Houdini 1.5a.
Everything falls nicely within the error margins of the gauntlets.

You've picked probably the only opponent against which Houdini 1.5a scored better than Houdini 3, and then discuss this result as if it was not a statistical outlier.

Robert

P.S. It always appears to be difficult to discuss these elementary statistics on computer chess forums... it's rather frustrating for me.

What i stated is clear and published by CEGT. Even not so many games have been played the result is anyway surprising, that' s pretty obvious.

You are trying to demonstrate what i didn't say or perhaps i didn't explain well my thoughts.

I stated that not so many games were played but the result is surprising anyway.

I expected you to say something more than talking about the error bar.

I remember you ran matches at long time control against Stockfish, Komodo and H2 but not Rybka 4.1. I remember that you talked about a good algorithm for long time control in H3.

If you really believe that is just a question of number of games and error bar if was you i would set up 2 matches at long time control H1.5a-rybka 4.1 and H3-rybka 4.1 4 cores each with 500 games each or more. You have the machine to do it. Me i haven't.
That would be very interesting.

Best Regards

Didn't H3 beat Rybka 4 fairly handily in the IPON test?

[Eelco de Groot]

It is not a matter of believing but in all likelihood a matter of statistics. No need for hocus pocus, no need for the argument that Rybka is much better at long timecontrols etc. It is not true, Junior is much better at long time controls. That is why it always wins at the ICGA championships.

Regards, Eelco

[Houdini]

I stated that not so many games were played but the result is surprising anyway.

I expected you to say something more than talking about the error bar.

If the results fall within the error margins of the test, why are you surprised?
Your surprise can only come from a lack of awareness of the error margins of the test.
At some point you have to start accepting the reality of engine match variability, and accept that the error margins that are computed have a practical relevance.

What more can be said? Rybka 4 was one of the 9 engines in the development and testing framework of Houdini 3, I haven't got the any indication that it would perform particularly well or poorly against Houdini 3.

Robert

[Laskos]

I stated that not so many games were played but the result is surprising anyway.

I expected you to say something more than talking about the error bar.

If the results fall within the error margins of the test, why are you surprised?

Because often the hardest thing to convince some people is that they see a 20% probability event. They are constantly amazed in front of 1/5 of the things happening around them.

From "The Power of Coincidence: Some Notes on "Psychic" Predictions" by Robert Novella:

There are many simple reasons why most people misinterpret coincidences:

-- Humans have a poor innate grasp of probability.
-- We believe that all effects must have deliberate causes.
-- We do not understand the laws regarding truly large numbers.
-- We easily succumb to selective validation—the tendency to remember only positive correlations and forget the far more numerous misses.


Kai

[Waschbaer]

I stated that not so many games were played but the result is surprising anyway.

I expected you to say something more than talking about the error bar.

I own a coin with 2 numbers, the number 1 on one side, the number 2 of the other side.
My friend came along with a cube, with the numbers 1 to 6 on it.
We played a match, I played with the coin, my friend with his cube.
I was told, the cube is able to show 3,5 points +/- 2.5 Points on every try, my coin only 1.5 points +/- 0.5 point on every move.
This numbers were figured out by 10000 test shots.

We did our test, 2 shots.
My friend with the cube got 2 and 1 points,
I got 2 and 2 point!!!

Now he is moaning,
how is that possible, ok, I know we did only 2 shots, but tell me, how it is possible the you bet me?
And please don`t tell me something about error margin,
you should explain what happend.
I'm sure now a coin always delivers higher numbers as a cube, ..., ... ...


Got it?

[Uri Blass]

It is not a matter of believing but in all likelihood a matter of statistics. No need for hocus pocus, no need for the argument that Rybka is much better at long timecontrols etc. It is not true, Junior is much better at long time controls. That is why it always wins at the ICGA championships.

Regards, Eelco

No
Junior won the ICGA championship because of other factors.
1)better book that is relevant for Junior style.
2)Houdini3 did not participate.
3)Rybka4 did not participate.

I saw no proof that Junior is better at long time control relative to houdini.