Deep Rybka 4.1 vs Houdini 3

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
S.Taylor
Posts: 8499
Joined: Thu Mar 09, 2006 2:25 am
Location: Jerusalem Israel

Re: Deep Rybka 4.1 vs Houdini 3

Post by S.Taylor » Mon Oct 22, 2012 3:55 pm

Houdini wrote:
MM wrote:I stated that not so many games were played but the result is surprising anyway.

I expected you to say something more than talking about the error bar.
If the results fall within the error margins of the test, why are you surprised?
Your surprise can only come from a lack of awareness of the error margins of the test.
At some point you have to start accepting the reality of engine match variability, and accept that the error margins that are computed have a practical relevance.

What more can be said? Rybka 4 was one of the 9 engines in the development and testing framework of Houdini 3, I haven't got the any indication that it would perform particularly well or poorly against Houdini 3.

Robert
I too, will always accept scientific/mathematical explanations. However i still like to see things the way i wished i could. And i like to know that a particular engine i want to choose, will give the best analysis vs any other engine or position i am interested that it should overcome. So if i make Houdini 3 my favorite engine, and want to see best refutations of Rybka 4.1 play, i may have to know that it is not Houdini 3 which gives it, but that i should have been happy 2 years earlier, with Houdini 1.5, because there was never to be a better engine for what i wanted to see. (If there will be one day, it could take much longer until it is created).

tomgdrums
Posts: 736
Joined: Wed Dec 02, 2009 8:48 am

Re: Deep Rybka 4.1 vs Houdini 3

Post by tomgdrums » Mon Oct 22, 2012 4:47 pm

Uri Blass wrote:
Eelco de Groot wrote:It is not a matter of believing but in all likelihood a matter of statistics. No need for hocus pocus, no need for the argument that Rybka is much better at long timecontrols etc. It is not true, Junior is much better at long time controls. That is why it always wins at the ICGA championships.

Regards, Eelco
No
Junior won the ICGA championship because of other factors.
1)better book that is relevant for Junior style.
2)Houdini3 did not participate.
3)Rybka4 did not participate.

I saw no proof that Junior is better at long time control relative to houdini.
Junior Won because:

1) Yes there were some definite heavyweights not in the competition

2) Junior is always capable of pulling a rabbit out of the hat in tournament type play.

They all have tuned books and a team of bookmakers.

I don't even look at Junior with regards to engine vs. engine competition.

If I had to pick only one other engine besides Houdini for post mortem analysis on my OTB games I would make Junior that second engine. It's evals are different and good and often educational and it works well with Houdini.

(Komodo 4 would be 3rd choice followed by Hiarcs and/or a new Shredder if it ever comes out)

MM
Posts: 766
Joined: Sun Oct 16, 2011 9:25 am

Re: Deep Rybka 4.1 vs Houdini 3

Post by MM » Mon Oct 22, 2012 9:05 pm

Houdini wrote:
MM wrote:I stated that not so many games were played but the result is surprising anyway.

I expected you to say something more than talking about the error bar.
If the results fall within the error margins of the test, why are you surprised?
Your surprise can only come from a lack of awareness of the error margins of the test.
At some point you have to start accepting the reality of engine match variability, and accept that the error margins that are computed have a practical relevance.

What more can be said? Rybka 4 was one of the 9 engines in the development and testing framework of Houdini 3, I haven't got the any indication that it would perform particularly well or poorly against Houdini 3.

Robert

If Federer loses at Wimbledon against Ferrer i would be surprised, even it's only a match.

If Barcelona loses against Malaga in the Liga i would be surprised.

If Houdini 3 loses against Crafty in an engine tour i would be surprised.

The surprise has nothing to do with the number of the games.

CEGT didn't run 6 games, they ran 50 games and 100 games. The error bar? Yes it exists but perhaps it could reverse in favor of Houdini 1.5a, couldn't it?

I mean, if the performance against Rybka 4.1 is 56% in favor of Houdini 3 and 62% in favor of Houdini 1.5a, after 50 games and 100 games, can someone expert in statistics tell me how much is the percentage of chances that Houdini1.5a performs better than Houdini 3 against Rybka 4.1? It's not a simple question of error bar, it's something more.

Anyway I didn't say that Houdini 1.5a 4 cpu is stronger than Houdini 3 4 cpu against Rybka 4.1 4 cpu.
I just said that i would be surprised if it was so, indications of CEGT are not good for Houdini 3 but not enough games have been played (and not enough games will be never played).


I hope some supercomputer owner will run a match at long time control because i remember TCEC of Marty Thoresen and it was a fantastic event, that Houdini won twice against Rybka.


Best Regards
MM

MM
Posts: 766
Joined: Sun Oct 16, 2011 9:25 am

Re: Deep Rybka 4.1 vs Houdini 3

Post by MM » Mon Oct 22, 2012 9:17 pm

tomgdrums wrote: Didn't H3 beat Rybka 4 fairly handily in the IPON test?
Yes but it was 5' + 3'', not long time control.

Best Regards
MM

User avatar
Dr.Wael Deeb
Posts: 9773
Joined: Wed Mar 08, 2006 7:44 pm
Location: Amman,Jordan

Re: Deep Rybka 4.1 vs Houdini 3

Post by Dr.Wael Deeb » Mon Oct 22, 2012 9:46 pm

MM wrote:
tomgdrums wrote: Didn't H3 beat Rybka 4 fairly handily in the IPON test?
Yes but it was 5' + 3'', not long time control.

Best Regards
I test at 20 minutes + 20 seconds increment time control on an overclocked i7 machine and I consider it rather a relatively fast time control.....

The IPON time contol is not my cup of tea....

It's a classic blitz time control.......
Dr.D
_No one can hit as hard as life.But it ain’t about how hard you can hit.It’s about how hard you can get hit and keep moving forward.How much you can take and keep moving forward….

MM
Posts: 766
Joined: Sun Oct 16, 2011 9:25 am

Re: Deep Rybka 4.1 vs Houdini 3

Post by MM » Mon Oct 22, 2012 9:56 pm

Dr.Wael Deeb wrote:
MM wrote:
tomgdrums wrote: Didn't H3 beat Rybka 4 fairly handily in the IPON test?
Yes but it was 5' + 3'', not long time control.

Best Regards
I test at 20 minutes + 20 seconds increment time control on an overclocked i7 machine and I consider it rather a relatively fast time control.....

The IPON time contol is not my cup of tea....

It's a classic blitz time control.......
Dr.D
40/120 repeating is cool.

Best Regards
MM

User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Deep Rybka 4.1 vs Houdini 3

Post by Laskos » Mon Oct 22, 2012 10:39 pm

MM wrote:

If Federer loses at Wimbledon against Ferrer i would be surprised, even it's only a match.

If Barcelona loses against Malaga in the Liga i would be surprised.

If Houdini 3 loses against Crafty in an engine tour i would be surprised.

The surprise has nothing to do with the number of the games.
I see, you seem to be often bewildered in life.


CEGT didn't run 6 games, they ran 50 games and 100 games. The error bar? Yes it exists but perhaps it could reverse in favor of Houdini 1.5a, couldn't it?

I mean, if the performance against Rybka 4.1 is 56% in favor of Houdini 3 and 62% in favor of Houdini 1.5a, after 50 games and 100 games, can someone expert in statistics tell me how much is the percentage of chances that Houdini1.5a performs better than Houdini 3 against Rybka 4.1? It's not a simple question of error bar, it's something more.
Strictly from those two matches, maybe 80% that H1.5 performs better than H3 against Rybka at LTC. From all the info available in all matches, it seems a 20% fluke, and it's still more probable that H3 performs better.

Anyway I didn't say that Houdini 1.5a 4 cpu is stronger than Houdini 3 4 cpu against Rybka 4.1 4 cpu.
I just said that i would be surprised if it was so, indications of CEGT are not good for Houdini 3 but not enough games have been played (and not enough games will be never played).


I hope some supercomputer owner will run a match at long time control because i remember TCEC of Marty Thoresen and it was a fantastic event, that Houdini won twice against Rybka.


Best Regards

User avatar
Houdini
Posts: 1471
Joined: Mon Mar 15, 2010 11:00 pm
Contact:

Re: Deep Rybka 4.1 vs Houdini 3

Post by Houdini » Mon Oct 22, 2012 10:40 pm

MM wrote:CEGT didn't run 6 games, they ran 50 games and 100 games. The error bar? Yes it exists but perhaps it could reverse in favor of Houdini 1.5a, couldn't it?
No, it could not!
Let me try to explain.

It was known before the Houdini 3 run started that Houdini 1.5a had scored a 3132 performance against Rybka 4, a 51 point over-performance compared to its 3081 rating.

Houdini 3's final average rating was 3151, only 19 Elo above this Houdini 1.5a performance.
In a 50-game match, there's about a 30% probability that an engine has a performance of at least 20 Elo less than its rating.
This means that even before the Houdini 3 run started, we knew that there was a 30% probability that Houdini 3 would score less against Rybka 4 than Houdini 1.5a had done in its extraordinary run.

So you're surprised at an event that had a 30% probability of occurring!

Basically you've picked the strongest Houdini 1.5a performance in any of its gauntlet matches, and then are surprised that Houdini 3 doesn't repeat this extraordinary performance against the same opponent.

Robert

S.Taylor
Posts: 8499
Joined: Thu Mar 09, 2006 2:25 am
Location: Jerusalem Israel

Re: Deep Rybka 4.1 vs Houdini 3

Post by S.Taylor » Tue Oct 23, 2012 12:18 am

Houdini wrote:
So you're surprised at an event that had a 30% probability of occurring!

Robert
If the quality of play (many of the moves) by Houdini 3 in those games is better, even in games that were ultimately drawn or lost, then that is still good news to me.

User avatar
Dr.Wael Deeb
Posts: 9773
Joined: Wed Mar 08, 2006 7:44 pm
Location: Amman,Jordan

Re: Deep Rybka 4.1 vs Houdini 3

Post by Dr.Wael Deeb » Tue Oct 23, 2012 8:01 am

MM wrote:
Dr.Wael Deeb wrote:
MM wrote:
tomgdrums wrote: Didn't H3 beat Rybka 4 fairly handily in the IPON test?
Yes but it was 5' + 3'', not long time control.

Best Regards
I test at 20 minutes + 20 seconds increment time control on an overclocked i7 machine and I consider it rather a relatively fast time control.....

The IPON time contol is not my cup of tea....

It's a classic blitz time control.......
Dr.D
40/120 repeating is cool.

Best Regards
Yes,but I have only two overclocked i7 systems for testing purposes and if I use longer time controls I need ages to finish my tournaments let alone to build a reliable rating list.........
Dr.D
_No one can hit as hard as life.But it ain’t about how hard you can hit.It’s about how hard you can get hit and keep moving forward.How much you can take and keep moving forward….

Post Reply