Deep Rybka 4.1 vs Houdini 3

[S.Taylor]

I stated that not so many games were played but the result is surprising anyway.

I expected you to say something more than talking about the error bar.

If the results fall within the error margins of the test, why are you surprised?
Your surprise can only come from a lack of awareness of the error margins of the test.
At some point you have to start accepting the reality of engine match variability, and accept that the error margins that are computed have a practical relevance.

What more can be said? Rybka 4 was one of the 9 engines in the development and testing framework of Houdini 3, I haven't got the any indication that it would perform particularly well or poorly against Houdini 3.

Robert

I too, will always accept scientific/mathematical explanations. However i still like to see things the way i wished i could. And i like to know that a particular engine i want to choose, will give the best analysis vs any other engine or position i am interested that it should overcome. So if i make Houdini 3 my favorite engine, and want to see best refutations of Rybka 4.1 play, i may have to know that it is not Houdini 3 which gives it, but that i should have been happy 2 years earlier, with Houdini 1.5, because there was never to be a better engine for what i wanted to see. (If there will be one day, it could take much longer until it is created).

[tomgdrums]

It is not a matter of believing but in all likelihood a matter of statistics. No need for hocus pocus, no need for the argument that Rybka is much better at long timecontrols etc. It is not true, Junior is much better at long time controls. That is why it always wins at the ICGA championships.

Regards, Eelco

No
Junior won the ICGA championship because of other factors.
1)better book that is relevant for Junior style.
2)Houdini3 did not participate.
3)Rybka4 did not participate.

I saw no proof that Junior is better at long time control relative to houdini.

Junior Won because:

1) Yes there were some definite heavyweights not in the competition

2) Junior is always capable of pulling a rabbit out of the hat in tournament type play.

They all have tuned books and a team of bookmakers.

I don't even look at Junior with regards to engine vs. engine competition.

If I had to pick only one other engine besides Houdini for post mortem analysis on my OTB games I would make Junior that second engine. It's evals are different and good and often educational and it works well with Houdini.

(Komodo 4 would be 3rd choice followed by Hiarcs and/or a new Shredder if it ever comes out)

[MM]

I stated that not so many games were played but the result is surprising anyway.

I expected you to say something more than talking about the error bar.

If the results fall within the error margins of the test, why are you surprised?
Your surprise can only come from a lack of awareness of the error margins of the test.
At some point you have to start accepting the reality of engine match variability, and accept that the error margins that are computed have a practical relevance.

What more can be said? Rybka 4 was one of the 9 engines in the development and testing framework of Houdini 3, I haven't got the any indication that it would perform particularly well or poorly against Houdini 3.

Robert

If Federer loses at Wimbledon against Ferrer i would be surprised, even it's only a match.

If Barcelona loses against Malaga in the Liga i would be surprised.

If Houdini 3 loses against Crafty in an engine tour i would be surprised.

The surprise has nothing to do with the number of the games.

CEGT didn't run 6 games, they ran 50 games and 100 games. The error bar? Yes it exists but perhaps it could reverse in favor of Houdini 1.5a, couldn't it?

I mean, if the performance against Rybka 4.1 is 56% in favor of Houdini 3 and 62% in favor of Houdini 1.5a, after 50 games and 100 games, can someone expert in statistics tell me how much is the percentage of chances that Houdini1.5a performs better than Houdini 3 against Rybka 4.1? It's not a simple question of error bar, it's something more.

Anyway I didn't say that Houdini 1.5a 4 cpu is stronger than Houdini 3 4 cpu against Rybka 4.1 4 cpu.
I just said that i would be surprised if it was so, indications of CEGT are not good for Houdini 3 but not enough games have been played (and not enough games will be never played).


I hope some supercomputer owner will run a match at long time control because i remember TCEC of Marty Thoresen and it was a fantastic event, that Houdini won twice against Rybka.


Best Regards

[MM]

Didn't H3 beat Rybka 4 fairly handily in the IPON test?

Yes but it was 5' + 3'', not long time control.

Best Regards

[Dr.Wael Deeb]

Didn't H3 beat Rybka 4 fairly handily in the IPON test?

Yes but it was 5' + 3'', not long time control.

Best Regards

I test at 20 minutes + 20 seconds increment time control on an overclocked i7 machine and I consider it rather a relatively fast time control.....

The IPON time contol is not my cup of tea....

It's a classic blitz time control.......
Dr.D

[MM]

Didn't H3 beat Rybka 4 fairly handily in the IPON test?

Yes but it was 5' + 3'', not long time control.

Best Regards

I test at 20 minutes + 20 seconds increment time control on an overclocked i7 machine and I consider it rather a relatively fast time control.....

The IPON time contol is not my cup of tea....

It's a classic blitz time control.......
Dr.D

40/120 repeating is cool.

Best Regards

[Laskos]

If Federer loses at Wimbledon against Ferrer i would be surprised, even it's only a match.

If Barcelona loses against Malaga in the Liga i would be surprised.

If Houdini 3 loses against Crafty in an engine tour i would be surprised.

The surprise has nothing to do with the number of the games.

I see, you seem to be often bewildered in life.

CEGT didn't run 6 games, they ran 50 games and 100 games. The error bar? Yes it exists but perhaps it could reverse in favor of Houdini 1.5a, couldn't it?

I mean, if the performance against Rybka 4.1 is 56% in favor of Houdini 3 and 62% in favor of Houdini 1.5a, after 50 games and 100 games, can someone expert in statistics tell me how much is the percentage of chances that Houdini1.5a performs better than Houdini 3 against Rybka 4.1? It's not a simple question of error bar, it's something more.

Strictly from those two matches, maybe 80% that H1.5 performs better than H3 against Rybka at LTC. From all the info available in all matches, it seems a 20% fluke, and it's still more probable that H3 performs better.

Anyway I didn't say that Houdini 1.5a 4 cpu is stronger than Houdini 3 4 cpu against Rybka 4.1 4 cpu.
I just said that i would be surprised if it was so, indications of CEGT are not good for Houdini 3 but not enough games have been played (and not enough games will be never played).


I hope some supercomputer owner will run a match at long time control because i remember TCEC of Marty Thoresen and it was a fantastic event, that Houdini won twice against Rybka.


Best Regards

[Houdini]

CEGT didn't run 6 games, they ran 50 games and 100 games. The error bar? Yes it exists but perhaps it could reverse in favor of Houdini 1.5a, couldn't it?

No, it could not!
Let me try to explain.

It was known before the Houdini 3 run started that Houdini 1.5a had scored a 3132 performance against Rybka 4, a 51 point over-performance compared to its 3081 rating.

Houdini 3's final average rating was 3151, only 19 Elo above this Houdini 1.5a performance.
In a 50-game match, there's about a 30% probability that an engine has a performance of at least 20 Elo less than its rating.
This means that even before the Houdini 3 run started, we knew that there was a 30% probability that Houdini 3 would score less against Rybka 4 than Houdini 1.5a had done in its extraordinary run.

So you're surprised at an event that had a 30% probability of occurring!

Basically you've picked the strongest Houdini 1.5a performance in any of its gauntlet matches, and then are surprised that Houdini 3 doesn't repeat this extraordinary performance against the same opponent.

Robert

[S.Taylor]

So you're surprised at an event that had a 30% probability of occurring!

Robert

If the quality of play (many of the moves) by Houdini 3 in those games is better, even in games that were ultimately drawn or lost, then that is still good news to me.

[Dr.Wael Deeb]

Didn't H3 beat Rybka 4 fairly handily in the IPON test?

Yes but it was 5' + 3'', not long time control.

Best Regards

I test at 20 minutes + 20 seconds increment time control on an overclocked i7 machine and I consider it rather a relatively fast time control.....

The IPON time contol is not my cup of tea....

It's a classic blitz time control.......
Dr.D

40/120 repeating is cool.

Best Regards

Yes,but I have only two overclocked i7 systems for testing purposes and if I use longer time controls I need ages to finish my tournaments let alone to build a reliable rating list.........
Dr.D