Uri Blass wrote:I also do not agree that today you can easily buy a quad core-3 box with 16 cores.
This is one of those times where I really don't care what _you_ believe. I care about what I _know_ to be true. You can order 'em from Dell. From IBM. From Sun. The list goes on and on and on.
I was pretty sure that these aren't available yet (NDA'd apparently). I was looking last week or so, and Intel hasn't released an i7 that supports multi-socket motherboards. Supposedly they will create a new socket type, like the core2's 771/775 (i7 is now 1366). If you know a link where one is available, it would be appreciated...
I wasn't particularly talking about quad I7's as there are only two so far and they are pretty new. But quad chip quad cores have been around a long time. And even 8 chip quad cores from Sun, et al...
Again, since I had originally suggested a PII/300 and I7 as late 1998 and late 2008 a single-chip is the most direct comparison staying away from exotica...
According to this the Pentium II/450 was introduced at the end of August 2008, so I expect some computers with the chip were available by the end of the year.
Based on my recollections, a Pentium/300 (not PII) should run Crafty at about 75M n/s. I think that's in rough agreement with Bob's claim of 100M n/s on a PII/300.
Dirt wrote:
According to this the Pentium II/450 was introduced at the end of August 2008, so I expect some computers with the chip were available by the end of the year.
Based on my recollections, a Pentium/300 (not PII) should run Crafty at about 75M n/s. I think that's in rough agreement with Bob's claim of 100M n/s on a PII/300.
You mean August 1998. And 75K n/s. I remeber, on my P200 Crafty was about 75K n/s, on PII 450 it could be 150K n/s. I think Uri's factor of 75 in 10 years is the best estimate. This, if taking into account the latest high-end boxes. But Rybka is able to beat Fritz 5 in 1:200 time controls. Ok, Bob tried to explain that hardware improvement leads to software improvement, but besides 64-bit, it is still software tuning to faster hardware.
This debate reminds me fondly of Proebsting's law... (paraphrased) "Compiler optimization advances double computing power every 18 years."
I guess we're assuming a hardware speedup of somewhere in the 170x-200x range over the last 10 years. I will be amazed if advances in chess programming techniques make as much difference as that hardware speedup does, even with the impressively low branching factors of today's best engines.
I'm hoping this experiment will show that the steady increase in engine playing strengths over the past 10 years is mostly the result of faster hardware. =)
I still believe the efforts of engine programmers are worthwhile, though. As the hardware performance increases, different software techniques seem to occupy the "sweet spot".
wgarvin wrote:
I guess we're assuming a hardware speedup of somewhere in the 170x-200x range over the last 10 years.
Where did you get 200x factor? Can you show me a mainstream box with Crafty at 20-30M n/s? I repeat, in Rybka (and Naum probably) case software gain is larger than the hardware one.
Don wrote:I'm sure the only 10 year time period that THEY would be interested in would have to include Rybka. I think most of their confidence is pinned on this single program and also their only hope, as slight as it is.
To me it's more reasonable to include a more representative 10 year period, not one that saw the introduction of an uncharacteristically strong program. For them, a bad 10 year period might just barely include the introduction of brand new hardware from Intel.
So the precise time period you settle on does matter - however I think 10 years is long enough to smooth out the bumps enough to not matter.
I'm not saying, and I think Uri has already disclaimed, that software improvements have always outpaced those of hardware. That this may be true mostly because of Rybka I don't deny. I do think that many people outside of the computer chess community underestimate the magnitude of the software improvements that have taken place over the years, which makes me a bit defensive on behalf of the software authors.
Yes, I agree completely on this. In the minds of many outsiders chess is a "brute force" hardware thing.
Rybka + today's hardware vs. Fritz 5 (top machine Dec 1998) + Today's hardware means an Elo difference of 622 points, to come back to something Uri mentioned or suggested.
That is 622 Elo points in exactly 10 years purely due to software. Can we say that hardware is granting the same amount?
If doubling the speed is 75 Elo points, 600 Elo points means 2^8 = 256 faster hardware needed to account for this. If Bob is right about 1:200 (which means doubling the speed every 1 year 3-4 months, beating moore's law) hardware is close to that. We can discuss 50 or 100 elo points more or less here and there, but the conclusion will be that the improvement due to software and hardware are equal or very close, going hand to hand.
Miguel
You guys always find a way to give Rybka a big advantage. I really don't believe a program developed 10 years ago should be expected to run well on hardware that will not exist for 10 years. Even if you just consider that it was compiled on a 10 year old compiler with the wrong optimizations for this platform and that it wasn't designed for 64 bit computers because there were not common enough, that's a pretty big hit. (I suppose you could consider compiler quality a software issue but not compiler based optimizations.)
I seriously doubt Fritz was tuned to run at a time control of 1 move every 10 hours either. You might claim this is a software issue but it's a tuning issue. Even with these factors I have no doubt that Rybka would still be much stronger.
Tuning, optimizations, etc. are not hardware so I guess I can include it in "software". Let's suppose that we allow Fritz to recompile with optimizations. Let's say it becomes twice faster => 75 Elo. Then the difference is 622-75 = 548 points. My conclusion does not change. Can hardware alone in 10 years justify ~550 Elo points. Yes, maybe around that number, but not much higher than that.
I gave you before a 1:200 speed up by hardware, but this is way above the 1:32 predicted by the Moore's law. By Moore's Law, 550 points is justified in about 15 years, not ten.
Moore's law can account for 375 Elo points in 10 years. Well, that is about the difference between Fritz 1999 and Fritz 2009 in current hardware. We do not need Rybka and we could use Naum.
I can't see that the numbers we have in hand back that hardware improvement was more important than software. In fact, the opposite may make more sense, IMHO. At best, both contributed it very much, in the same ball park.
Miguel
And despite the math you are using I do not see Rybka 1 second beating Fritz at 200 seconds. I would definitely have to see this.
Let me remind you again that Moore's law says _nothing_ about processor speed. It says the _density_ doubles every 2 years or so, letting us put twice as many transistors on a chip. That may well translate into a speedup of far more than 2. For example multiple cores, multiple pipelines, larger L1/L2 cache, better memory controller, etc...
bob wrote:First let's settle on a 10 year hardware period. The q6600 is two years old. If you want to use that as a basis, we need to return to early 1997 to choose the older hardware. The Pentium 2 (Klamath) came out around the middle of 1997, which probably means the best was the Pentium pro 200. I suspect we are _still_ talking about 200:1
This is not about simple clock frequency improvements, more modern architectures are faster for other reasons such as better speculative execution, more pipelines, register renaming, etc...
Correct me if I'm wrong, but in moving to a time handicap you seem to be ignoring the parallel search inefficiency we were both just explaining to Louis Zulli. Shouldn't that be taken into account?
None of this will matter unless it's really a close match - so I would be prepared to simple test single processor Rybka vs whatever and see what happens. If Rybka loses we have a "beta cut-off" and can stop, otherwise we must test something a little more fair and raise alpha.
If the parallel search overhead means that the ratio should really be, say, 150:1 then I don't think Rybka losing really proves your point. If there should be such a reduction, and how large it should be, is a question I am asking.
So if Rybka loses with say a 32 to 1 handicap you are saying that we should give her even less time to see if she still loses?
This is going around in circles. It is easy to quantify the hardware. I'd suggest taking the best of today, the intel I7 (core-3) and the best of late 1998. Limit it to a single chip for simplicity, but no limit on how many cores per chip. I believe this is going to be about a 200:1 time handicap to emulate the difference between the 4-core core-3 from intel and the best of 1998, which was the PII/300 processor.
For comparison, crafty on a quad-core I7 runs at 20M nodes per second, while on the single-cpu PII/300 was running at not quite 100K nodes per second. A clean and simple factor of 200x faster hardware over that period (and again, those quoting moore's law are quoting it incorrectly, it does _not_ say processor speed doubles every 2 years, it says _density_ doubles every 2 years, which is a different thing entirely). Clock speeds have gone steadily upward, but internal processor design has improved even more. Just compare a 2.0ghz core2 cpu against a 4.0ghz older processor to see what I mean.)
so that fixes the speed differential over the past ten years with high accuracy. Forget the discussions about 50:1 or the stuff about 200:1 being too high. As Bill Clinton would say, "It is what it is." And what it is is 200x.
That is almost 8 doublings, which is in the range of +600 Elo. That is going to be a great "equalizer" in this comparison. 200x is a daunting advantage to overcome. And if someone really thinks software has produced that kind of improvement, we need to test it and put it to rest once and for all...
I will accept that a program today running on 4 cores will see some overhead due to the parallel search. But I don't think it is worth arguing about whether we should scale back the speed because of the overhead. That is simply a software issue as well, as it is theoretically possible to have very little overhead. If the software can't quite use the computing power available, that is a software problem, not a hardware limit.
Then you have to accept that Fritz 5 is 622 Elo points below Rybka in current hardware. That is a bit more than the 600 points you estimate harwdare provided in 10 years.
Miguel
I don't accept that at all. That's why I suggested we run a test rather than using ratings that are very old and out of date. how many games has fritz 5.32 played _recently_ on the rating lists? That makes a huge difference and it might be better now since it is still going to beat the top programs on occasion, and with them so much higher its rating would likely drag up as well.
What do you find so wrong about these data?
Simple. Fritz 5.32 played on the SSDF 10 years ago. How many games has it played in the last 5 years? As many in the past 2-3 years as Rybka? If not, its rating will be depressed, since the old ratings tend to be static and the newer programs are advancing onward thanks to both hardware and software improvements. You can no more compare 1998 Fritz rating to 2008 Rybka rating than you can compare Capablanca's rating to Kasparov's to decide who would be better... Kasparov is not _that_ much better than Capa... Yet in Capa's time 2800+ never happened.
CCRL 40/4 Rating List - Custom engine selection
388092 games played by 744 programs, run by 12 testers
Ponder off, General books (up to 12 moves), 3-4-5 piece EGTB
Time control: Equivalent to 40 moves in 4 minutes on Athlon 64 X2 4600+ (2.4 GHz)
Computed on January 10, 2009 with Bayeselo based on 388'092 games
Tested by CCRL team, 2005-2009, http://computerchess.org.uk/ccrl/404/
Rank Engine ELO + - Score AvOp Games
1 Fritz 5.32 2642 +13 -13 53.2% -24.5 2132
Miguel
So let's run the test rather than speculating...
I have some Crafty versions that should be right for that time frame. Crafty 15.0 was the first parallel search version. I suspect something in the 16.x versions or possibly 17.x versions was used at the end of 1998. Crafty ran on a quad pentium pro early in 1998 whe
n version 15.0 was done...
Dirt wrote:
According to this the Pentium II/450 was introduced at the end of August 2008, so I expect some computers with the chip were available by the end of the year.
Based on my recollections, a Pentium/300 (not PII) should run Crafty at about 75M n/s. I think that's in rough agreement with Bob's claim of 100M n/s on a PII/300.
You mean August 1998. And 75K n/s. I remeber, on my P200 Crafty was about 75K n/s, on PII 450 it could be 150K n/s. I think Uri's factor of 75 in 10 years is the best estimate. This, if taking into account the latest high-end boxes. But Rybka is able to beat Fritz 5 in 1:200 time controls. Ok, Bob tried to explain that hardware improvement leads to software improvement, but besides 64-bit, it is still software tuning to faster hardware.
Kai
I ran on a PII/400 quad for a couple of years. I was getting about 100K. That was on a PII/400 xeon with 2mb of L2 cache, which was faster than the usual PII/450 single-cpu box that came in a desktop. The quad had 4-way memory interleaving to further improve memory bandwidth. I didn't get the quad until the middle of 1999 however. If you prefer to use that particular processor, then 100K is a real good estimate. If you want to take that up to 125K that's OK. I know that on today's latest, as of December 2008, I could hit 20M on a single chip quad-core. I get a factor of 160X faster. If you want to account for 4-processor search overhead, using the 3.3x speedup numbers from a long quad-cpu test I discussed and which we analyzed here on CCC, then we change that to 132:1...
wgarvin wrote:
I guess we're assuming a hardware speedup of somewhere in the 170x-200x range over the last 10 years.
Where did you get 200x factor? Can you show me a mainstream box with Crafty at 20-30M n/s? I repeat, in Rybka (and Naum probably) case software gain is larger than the hardware one.
Kai
Intel I7 quad-core. 20M nodes per second. I ran on one of these ND'd middle of the year last year. They are now out and others are reporting the same number.
A dual-chip version will hit 40M. Again verified already. And I have run on prototype quad-core versions that scale perfectly in terms of NPS. hardware is still getting faster, even though the clock speeds are not advancing very much of late.
I realize that ego makes everyone want to _think_ the programming is more important. But it is not. It is important, to be sure, but the hardware advances have far outstripped the programming advances, and most that have been doing this for a good while realize this, even if they don't want to admit it.
I moved from a state-of-the-art mainframe computer in 1981 and by 1983 was the world computer chess champion. All because of the Cray, which was faster than anything else on the planet by a big margin... No programming breakthroughs. Just a solid-playing program that took advantage of some specific architectural features on the Cray (vectors primarily, but 64 bit words helped a lot as well) to do things I could not do on traditional mainframes because the cost was too high.
CRoberson wrote:While at the 2008 ACCA Pan American Computer Chess Championships,
Bob claimed he didn't believe software played a serious role in all the
rating improvements we've seen. He thought hardware deserved the
credit (assuming I understood the statement correctly. We were jumping
across several subjects and back that night.).
I beleive software has had much to do with it for several reasons.
I will start with one. The EBF with only MiniMax is 40. With Alpha-Beta
pruning, it drops to 6. In the early 1990's, the EBF was 4. Now, it is 2.
Dropping the EBF from 2 to 4 is huge. Lets look at a 20 ply search.
The speedup of EBF=2 vs EBF=4 is:
4^20/2^20 = 2^20 = 1,048,576
So, that is over a 1 million x speed up. Has hardware produced that much
since 1992?
Also, I believe eval improvements have caused an improvement in
rating scores.
An example of nonhardware improvements is on the SSDF rating list.
Rybka 1.0 beta score 2775 on a 450 MHz AMD.
Another not very serious thought:
A new piece of good chess software (a new commercial engine)
costs about 100 USD or EUR.
A new PC costs about 1000$.
How much ELO/$ do I get? How should I invest money
when I have a ten year old PC and engine?
Would Bob sell me his famous cluster for 100$?
Harald
You are thinking backward. Look at what you could get for $500 10 years ago, and what you get today, in terms of processor speed. Huge jump. I believe bigger than what you would get if you could come from ten years back thru time travel to buy a program today and go back again and run it on 10 year old hardware.
