Uri Blass wrote:
I am clearly interested in a fair test.
Note that I suggested to give Crafty an octal and to give Rybka a quad.
That's the part that I'm not sure is fair. Why don't we just limit this to what was available on a single chip and not consider what is possible with exotic configurations that almost nobody can afford for home use unless they are wealthy or fanatical (like we are.) Today 4 core is very common and back then even 2 processor machines were quite uncommon and required extra boards and such - I really don't think you get to have another 2 to 1 advantage and call it equitable.
I have no objection to give both ssdf hardware and the advantage from P200 single processor to Q6600(4 processors) is not close to 200:1 based on my knowledge(I guess that it may be 50:1 or at most 100:1).
Ok, I think that is fair and nobody should complain too much.
odd match may be interesting.
I feel sure that even the 32 bit version of rybka can win match with 32:1 time handicap(assuming not very fast time control) because older rybka2.3.2a 32 bits(weaker than rybka3) could win a match of 100 games against movei(stronger than fritz5.32) with 100:7 time handicap.
I am not sure about results with significantly bigger time handicap and 200:1 that bob suggest may be too big handicap.
Uri
His number seems big to me, but I'm not in a position to argue with him, he is probably correct about this. He also could have sped the program up since then and perhaps it's really more like 100 to 1 or 150 to 1?
At this point I would rather do it than continue to talk about it. I would prefer to avoid too much extrapolation and really use the programs we are talking about instead of extrapolating from Crafty, but this may be a good practical solution.
bob wrote: Old on new hardware will be the most interesting as I would not be surprised at all to see the old program beat R3 with R3 on old hardware and old program on new hardware...
That's what I expect to see - but there is a big question mark about whether the old program can actually utilize the new hardware. Your test would help sort that out and I think it is a good test to do.
I expect rybka to win at 120/40 time control even if we replace old programs by Crafty21.5 that has a similiar level to top programs of 1999(assuming that you give rybka the best hardware of 1999 that is a quad and give Crafty21.5 the best octal of today(I assume that programs of 1999 could not use efficiently something better than an octal).
Note that I suggest to use contempt=0 for rybka(the default contempt=15 is better against significantly weaker opponent but when the target is to win a match it is better to use contempt=0 based on the results that I read)
Uri
I have to point out that this is not a fair test the way you present it for several reasons. I knew from the beginning that it was stacked very much in favor of Rybka but I think you are trying to stack it more. I noticed you are now talking about 1999, not 1998 but let's say January of 1999 to be more precise. Also, even though quads may have been available, nobody was testing on quads back then. Everybody today it seems has a quad in their home (at least serious chess enthusiasts) and the ratings lists are testing with quads. Were quads being tested on the ratings lists in 1998?
I would agree to using the best of what the testing agencies were using at the time. I think that limits todays hardware to a quad (even though we can go much higher) and 10 years ago hardware to a single processor machine even though you could go higher.
Of course you would like to ramp up the hardware on both ends as much as possible because you realize that modern programs were designed for better hardware (which is part of MY argument.) So this would not contribute to making your point and you should not want this. Are you looking for the truth, or just to trying to construct a match that you can win? If you really want to know the truth here you cannot keep pushing for every possible advantage. This would become like one of those battle of the sexes tennis matches where the female is given all kinds of advantages because we all know this isn't really about whether men or women play better.
Rybka is very strong - we should both be interested in a fair test, not trying to nitpick every possible advantage to stack the odds in our favor - that wouldn't prove anything. If the test is too obviously unfair it just makes the results meaningless and people will debate it. (I'm sure they will anyway, but let's not make it too obvious that we secretly want to have more reasons to worship and adore Rybka.)
Of course it's clear that we cannot possibly construct a perfectly fair test but don't we need to be reasonable?
This is partly why I want to just do a prelim odds match - just to see if there is anything to talk about.
I am clearly interested in a fair test.
Note that I suggested to give Crafty an octal and to give Rybka a quad.
I have no objection to give both ssdf hardware and the advantage from P200 single processor to Q6600(4 processors) is not close to 200:1 based on my knowledge(I guess that it may be 50:1 or at most 100:1).
odd match may be interesting.
I feel sure that even the 32 bit version of rybka can win match with 32:1 time handicap(assuming not very fast time control) because older rybka2.3.2a 32 bits(weaker than rybka3) could win a match of 100 games against movei(stronger than fritz5.32) with 100:7 time handicap.
I am not sure about results with significantly bigger time handicap and 200:1 that bob suggest may be too big handicap.
Uri
First let's settle on a 10 year hardware period. The q6600 is two years old. If you want to use that as a basis, we need to return to early 1997 to choose the older hardware. The Pentium 2 (Klamath) came out around the middle of 1997, which probably means the best was the pentium pro 200. I suspect we are _still_ talking about 200:1
This is not about simple clock frequency improvements, more modern architectures are faster for other reasons such as better speculative execution, more pipelines, register renaming, etc...
bob wrote:
First let's settle on a 10 year hardware period. The q6600 is two years old. If you want to use that as a basis, we need to return to early 1997 to choose the older hardware. The Pentium 2 (Klamath) came out around the middle of 1997, which probably means the best was the pentium pro 200. I suspect we are _still_ talking about 200:1
This is not about simple clock frequency improvements, more modern architectures are faster for other reasons such as better speculative execution, more pipelines, register renaming, etc...
I'm sure the only 10 year time period that THEY would be interested in would have to include Rybka. I think most of their confidence is pinned on this single program and also their only hope, as slight as it is.
To me it's more reasonable to include a more representative 10 year period, not one that saw the introduction of an uncharacteristically strong program. For them, a bad 10 year period might just barely include the introduction of brand new hardware from Intel.
So the precise time period you settle on does matter - however I think 10 years is long enough to smooth out the bumps enough to not matter.
bob wrote:
First let's settle on a 10 year hardware period. The q6600 is two years old. If you want to use that as a basis, we need to return to early 1997 to choose the older hardware. The Pentium 2 (Klamath) came out around the middle of 1997, which probably means the best was the pentium pro 200. I suspect we are _still_ talking about 200:1
This is not about simple clock frequency improvements, more modern architectures are faster for other reasons such as better speculative execution, more pipelines, register renaming, etc...
I'm sure the only 10 year time period that THEY would be interested in would have to include Rybka. I think most of their confidence is pinned on this single program and also their only hope, as slight as it is.
To me it's more reasonable to include a more representative 10 year period, not one that saw the introduction of an uncharacteristically strong program. For them, a bad 10 year period might just barely include the introduction of brand new hardware from Intel.
So the precise time period you settle on does matter - however I think 10 years is long enough to smooth out the bumps enough to not matter.
Would be a better test to back up to right before Rybka "took the world by storm" and pick the best program of that year, and the hardware from that year, and then back up 10 years and pick the best hardware back then along with the same program. Now you can measure actual software improvement over 10 years and actual hardware improvement over the same 10 years...
Using Rybka wrecks the comparison since we do not know its roots or anything about what goes on inside. And since it wasn't available 10 years ago there is no measurement to make in that regard.
bob wrote:First let's settle on a 10 year hardware period. The q6600 is two years old. If you want to use that as a basis, we need to return to early 1997 to choose the older hardware. The Pentium 2 (Klamath) came out around the middle of 1997, which probably means the best was the pentium pro 200. I suspect we are _still_ talking about 200:1
This is not about simple clock frequency improvements, more modern architectures are faster for other reasons such as better speculative execution, more pipelines, register renaming, etc...
Correct me if I'm wrong, but in moving to a time handicap you seem to be ignoring the parallel search inefficiency we were both just explaining to Louis Zulli. Shouldn't that be taken into account?
bob wrote:First let's settle on a 10 year hardware period. The q6600 is two years old. If you want to use that as a basis, we need to return to early 1997 to choose the older hardware. The Pentium 2 (Klamath) came out around the middle of 1997, which probably means the best was the pentium pro 200. I suspect we are _still_ talking about 200:1
This is not about simple clock frequency improvements, more modern architectures are faster for other reasons such as better speculative execution, more pipelines, register renaming, etc...
Correct me if I'm wrong, but in moving to a time handicap you seem to be ignoring the parallel search inefficiency we were both just explaining to Louis Zulli. Shouldn't that be taken into account?
None of this will matter unless it's really a close match - so I would be prepared to simple test single processor Rybka vs whatever and see what happens. If Rybka loses we have a "beta cut-off" and can stop, otherwise we must test something a little more fair and raise alpha.
bob wrote:First let's settle on a 10 year hardware period. The q6600 is two years old. If you want to use that as a basis, we need to return to early 1997 to choose the older hardware. The Pentium 2 (Klamath) came out around the middle of 1997, which probably means the best was the Pentium pro 200. I suspect we are _still_ talking about 200:1
This is not about simple clock frequency improvements, more modern architectures are faster for other reasons such as better speculative execution, more pipelines, register renaming, etc...
Correct me if I'm wrong, but in moving to a time handicap you seem to be ignoring the parallel search inefficiency we were both just explaining to Louis Zulli. Shouldn't that be taken into account?
None of this will matter unless it's really a close match - so I would be prepared to simple test single processor Rybka vs whatever and see what happens. If Rybka loses we have a "beta cut-off" and can stop, otherwise we must test something a little more fair and raise alpha.
If the parallel search overhead means that the ratio should really be, say, 150:1 then I don't think Rybka losing really proves your point. If there should be such a reduction, and how large it should be, is a question I am asking.
bob wrote:First let's settle on a 10 year hardware period. The q6600 is two years old. If you want to use that as a basis, we need to return to early 1997 to choose the older hardware. The Pentium 2 (Klamath) came out around the middle of 1997, which probably means the best was the pentium pro 200. I suspect we are _still_ talking about 200:1
This is not about simple clock frequency improvements, more modern architectures are faster for other reasons such as better speculative execution, more pipelines, register renaming, etc...
Correct me if I'm wrong, but in moving to a time handicap you seem to be ignoring the parallel search inefficiency we were both just explaining to Louis Zulli. Shouldn't that be taken into account?
None of this will matter unless it's really a close match - so I would be prepared to simple test single processor Rybka vs whatever and see what happens. If Rybka loses we have a "beta cut-off" and can stop, otherwise we must test something a little more fair and raise alpha.
I didn't say this well. Let me try again.
If we do a simple time handicap game we just start in single processer mode for everybody and make Rybka accept a relatively modest time handicap of say 32 to 1. If this is a close match or Rybka loses, then there is not much point in continuing as it will only get worse for her. Otherwise, we try something a little less blatantly stacked in Rybka's favor but that is also fair. We can use the result of that match to give us an idea of what to try next.
I agree with bob that we can estimate the advantage of having more than 1 processor and am willing to fudge in the direction of being generous to Rybka so that this is not too boring.
Don wrote:I'm sure the only 10 year time period that THEY would be interested in would have to include Rybka. I think most of their confidence is pinned on this single program and also their only hope, as slight as it is.
To me it's more reasonable to include a more representative 10 year period, not one that saw the introduction of an uncharacteristically strong program. For them, a bad 10 year period might just barely include the introduction of brand new hardware from Intel.
So the precise time period you settle on does matter - however I think 10 years is long enough to smooth out the bumps enough to not matter.
I'm not saying, and I think Uri has already disclaimed, that software improvements have always outpaced those of hardware. That this may be true mostly because of Rybka I don't deny. I do think that many people outside of the computer chess community underestimate the magnitude of the software improvements that have taken place over the years, which makes me a bit defensive on behalf of the software authors.
bob wrote:First let's settle on a 10 year hardware period. The q6600 is two years old. If you want to use that as a basis, we need to return to early 1997 to choose the older hardware. The Pentium 2 (Klamath) came out around the middle of 1997, which probably means the best was the Pentium pro 200. I suspect we are _still_ talking about 200:1
This is not about simple clock frequency improvements, more modern architectures are faster for other reasons such as better speculative execution, more pipelines, register renaming, etc...
Correct me if I'm wrong, but in moving to a time handicap you seem to be ignoring the parallel search inefficiency we were both just explaining to Louis Zulli. Shouldn't that be taken into account?
None of this will matter unless it's really a close match - so I would be prepared to simple test single processor Rybka vs whatever and see what happens. If Rybka loses we have a "beta cut-off" and can stop, otherwise we must test something a little more fair and raise alpha.
I didn't say this well. Let me try again.
If we do a simple time handicap game we just start in single processor mode for everybody and make Rybka accept a relatively modest time handicap of say 32 to 1. If this is a close match or Rybka loses, then there is not much point in continuing as it will only get worse for her. Otherwise, we try something a little less blatantly stacked in Rybka's favor but that is also fair. We can use the result of that match to give us an idea of what to try next.
I agree with bob that we can estimate the advantage of having more than 1 processor and am willing to fudge in the direction of being generous to Rybka so that this is not too boring.
I see. I agree 32 to 1 is almost surely too little.
