harware vs software advances

[bob]

By the way, please tell me why we have to pretend parallel programs did not exist in the mid 90's?

Are we reading the same posts? Bob was accusing you of treating parallel search as something new since 1995. Now you are accusing him of the same thing?

Show me where I said that.

Quoted above: "please tell me why we have to pretend parallel programs did not exist in the mid 90's?"

He went to great pains to say exactly the opposite.

My point is that if we are comparing actual HARDWARE advances since the mid 90's and clearly parallel programs existed, then it's blatantly unfair to disallow parallel technology from the 90's in this test but eagerly embrace parallel technology from 2010.

Are we actually interested in seeing how much we advanced or not? Or is that you guys only live in the Intel PC world? This discussion isn't about PC's and how much the Pentium has advanced, it's about hardware advancement in general. I was using 1824 processors long ago and 4 processor alpha's in the 90's.

I might be getting this wrong but I think Bob was using parallel processing in the late 80's! But running on 4 processors in 2010 is a huge advancement? This is totally not the case and I don't see how this trivial point escapes you.

If Bob is really interesting in measure ACTUAL hardware improvement we should be comparing the best parallel alpha system running in 1995 to the best chip we have running in 2010, the i7-980x which is a real product.

Bob, are you interested in a fair comparison or do you just want to compare the highly inferior Intel product to the very best chip we have today?

The alpha systems cost more than the Pentiums, but they were affordable (as high end systems go.)

It would be _very_ difficult to locate 1995 hardware. And if you venture that far out, the alpha sucks. The Cray T932 was king-of-the-hill in 1995. But you aren't going to find one laying around. The alpha is gone. Pentium 90 to i7 represents at least a homogeneous series of processors. I just happened to have data on the p5/133, and it is trivial to get numbers for the i7 since many have them.

I don't believe it makes sense to take a processor that is not available today, and which was discontinued many years ago, and include that. Yes the alpha was better than intel at the time. But the newer Intels are far superior. With Intel we can at least compare a chip that was available in 1995 and compare it to chips available today. And to be honest, hardly anybody cares about anything but the PC platform since that is what 99.999999999% of the people are using to play chess. And they have not even seen the exotic hardware some of us used to use. All the rating lists use Intel (or AMD, same thing). There is better hardware available today than a single 6-core i7 as well. But very few programs run on clusters, and hardly anyone has access to the really high-end clusters from IBM and others.

BTW my first parallel search ran in 1978 at an ACM event using a dual-CPU univac box. I was not the first. Monty Newborn was using a group of data general machines connected by a channel-to-channel adapter even earlier.

As far as your alpha question goes, how can we use it? Development was discontinued 10+ years ago. Where would it be today if we had a 21564, 21664, etc? No way to know. The alpha was not particularly superior to Intel at the time, if you exclude the obvious 64 bit advantage it had for bitboards. Bruce ran on a cryo-version at the 1997 WMCCC, and reported that for ferret, the only advantage was the nearly 800mhz clock speed for the nitrogen-cooled processor. I saw more than that, but then I was 64 bits and Bruce was not. The alpha was pretty good for floating point, which was its major market in the scientific workstation market. Didn't help chess at all in that regard.

[bob]

By the way, please tell me why we have to pretend parallel programs did not exist in the mid 90's?

Are we reading the same posts? Bob was accusing you of treating parallel search as something new since 1995. Now you are accusing him of the same thing?

Show me where I said that.

Quoted above: "please tell me why we have to pretend parallel programs did not exist in the mid 90's?"

He went to great pains to say exactly the opposite.

My point is that if we are comparing actual HARDWARE advances since the mid 90's and clearly parallel programs existed, then it's blatantly unfair to disallow parallel technology from the 90's in this test but eagerly embrace parallel technology from 2010.

Are we actually interested in seeing how much we advanced or not? Or is that you guys only live in the Intel PC world? This discussion isn't about PC's and how much the Pentium has advanced, it's about hardware advancement in general. I was using 1824 processors long ago and 4 processor alpha's in the 90's.

I might be getting this wrong but I think Bob was using parallel processing in the late 80's! But running on 4 processors in 2010 is a huge advancement? This is totally not the case and I don't see how this trivial point escapes you.

If Bob is really interesting in measure ACTUAL hardware improvement we should be comparing the best parallel alpha system running in 1995 to the best chip we have running in 2010, the i7-980x which is a real product.

Bob, are you interested in a fair comparison or do you just want to compare the highly inferior Intel product to the very best chip we have today?

The alpha systems cost more than the Pentiums, but they were affordable (as high end systems go.)

A lot of poeple will naturally be interested in intel, since thats the platform they have played with in their own basement. And that's what crafty typically ran on in 1995.

This might be interesting as a DIFFERENT issue, but it's not representative of hardware advances in actual Desktop computers. Nobody ever said this a Pentium contest.

Then let's draw a line _somewhere_. That old 1995 Cray T90 will smoke a PC today with memory bandwidth. In fact, in 1995 we hit 7M nps. I have hit 100M on a hot Intel box. So hardware has improved 10x or so? Really?

Alpha is impossible to evaluate. The 1995 alphas were not particularly good. I believe the 21164 came out in either late 1996 or early 1997, so we had the "adequate" 21064 in 1995. Not particularly fast (we had one in fact and my chess program ran no faster on it than on Intel, until the 21264 came out which is where the alpha began to shine).

You will be hard-pressed to find a 21064 box to get any benchmark data on. I had a 164 that Dec gave me when I was helping them test their parallel C compiler stuff, and then ran on a 21264 cluster-like alpha, which was the target of that compiler. But I did not save anything from that box, and it would be the wrong processor anyway since it wasn't the 064.

Excepting a very "few" of us, most consider Intel to be the "only" computer. And it certainly is the only processor that survived from 1995 to 2010. If we venture outside Intel, we open _another_ pandora's box. For example, why do we only consider a single-chip i7 as the best of today? We've been able to buy dual-chip, quad-chip and even 8-way chip boxes for 10 years now. And some (Sequent, Sun, et. al.) make boxes (true SMP) with way more than that. Do those not exist?

So a simple "best single-chip" in 1995 vs the same for 2010 is at least something that can be measured.

What he's doing now with new and old crafty will give some estimate on the benefits made in intel advances. It seems to me it's boiling down to NPS comparisons anyway, old versus new, from HW improvements, and then the delta will be software. That's what you're really after.

That's not what I'm after and it's not about NPS as this is an extremely unreliable way to estimate what a program really does. It might be roughly ok but it can be off by a doubling or more.

Now if one could slow down an old program and a new program on new hardware (introducing delay loops to lower average NPS), then test them on a cluster, you could calc the ELO on 1995 equivalent hardware. Take your pick of 1995 performance by dialing down the NPS to the 1995 level that hardware was recorded as producing at the time with that program. Don't know how hard that would be.

The issue is that new program and old programs will not respond the same. I think it's roughly the same, but on a P90 rebel is 100 to 1 slower than on modern hardware, Bob thinks the "right" number is 1000 to 1.

This would imply that Bob could take the modern Crafty source code, go back in time to the P90 days, and get a 10 to 1 improvement just by rewriting the move generator. What he implies is so ridiculous as to be laughable.

You might get 2 or 3 times speedup if you start with a modern program that blows out the cache of the P90 or something but I did the reference test with an OLD program so that slowing it down 100 to 1 on new hardware (even a modern program would be reasonable.)

Maybe an x86 emulator could be hacked to accurately dial-down the effective Mhz. Then the old and new programs could be dialed down to 1995 speeds, but still tested on a cluster with thousands of games. Probably a tall order.

I think there is actually a utility for DOS that does something like this.

[bob]

To compare software, we need a 1995 program and 1995 hardware, and then the same 1995 program on 2010 hardware. But it doesn't need to be a lousy example of a 1995 program. Parallel search was old news in 1995. Genius never had that. So run it today and you immediately throw away 5/6 of the hardware advantage of today. Not reasonable, IMHO.

I think that it is not reasonable to take best of both worlds.

The combination of Genius and parallel search does not exist as software
so if you take parallel search as software from 1995 it means that you cannot take Genius3 and you need to find a program from 1995 when the programmer already used parallel search.

that's exactly what I have been saying...

It means that you can use Crafty of 1995 that is weaker than Genius of 1995 if you want to give the old program time advantage of 1000:1

Actually Crafty was not really weaker than genius in 1995. If you back up to the very early 90's (before Crafty) genius was a significant threat. But by 1995 I don't remember Genius really being an issue on chess servers or in tournaments...

I believe that Genius was better than Crafty on 1 cpu in 1995.
Genius was clearly the ssdf leading program in 1995 when ssdf used p90 for testing at 120/40 time control.

All I can say is that I didn't have exotic hardware in 1995, nor in 1996. At the 1996 WMCCC two versions of Crafty finished in 3-4 places. And playing on ICC, genius was a good program, but not an exceptional one. 2-3-4 years earlier, it was quite good compared to everyone else. But not in 1995-1996.

[bob]

Because in 1995 I had a _better_ parallel search implementation than I do today. In fact, in 1989 I had that _same_ "better implementation."

Random questions of the day for Dr. Hyatt:

Since you had a better parallel search implementation 15+ years ago, you had to have changed it somewhere in between then and now.

First question: Why did you change implementations?

I have already answered this. DTS was done in 1985-1988. The one unique characteristic was that it depended on an iterated search (loop increasing ply rather than recursive calls to increment ply). That lets you split anywhere in the tree. And it makes for a very messy search that is significantly harder to understand and modify. When I wrote Crafty, I chose to use the recursive negamax search implementation because of the clean way it expresses alpha/beta. And when I did the parallel search (work started about the same time that I started working on Crafty since I had plans to run Crafty on the Cray just as I was running Cray Blitz on that parallel hardware) I chose to keep the recursive negamax search. Which means no DTS. I backed up a bit to use something that was somewhat simpler, knowing that I was not going to see 32 CPU machines (like the Cray T932) for years. DTS improves with more processors.

So the why was to keep the code clean.

and

Second question: If you changed back to the "better implementation" now with current Crafty, would current Crafty's play improve?

[/quote]

It would be somewhat faster, because the CB parallel search performed better. But for 4 cores, the difference is not large. It gets larger as the number of processors climbs. I decided early on that I was not going to really target "exotic" hardware. I'd been doing that for 15 years at the time (we moved to the Cray in 1980) and getting access for testing was tough, it was hard enough to get access to compete. So I had decided to do something that better fit the "cheap machines" so that machine time would not be as big an issue.

A couple have taken my dissertation and implemented most of DTS. I think Zach did it in ZCT (he can respond if he wants). Ditto for Cozzie. Vincent asked a zillion questions about it, not sure what parts he used.

[rbarreira]

Just curious, what was the price of that 1995 Cray T90?

[bob]

I'm pretty sure the Pentium 90 was the fastest processor at that time. Intel introduced the Pentium Pro family (to which the P200 belongs) in late 1995.

Yes, I'll assume that circa 1994 CG3 was running on the P90.

Any idea of the speed? I don't suppose anybody has a Pentium 90 laying around do they?

Don

I have an operational Pentium 90.

Good. What we need is a comparison on 2 different platforms, the P90 and the newest thing you have.

Are you willing to try the following experiment: ?

Find the oldest Crafty you can that runs on both platforms and run some kind of time test on both machines. Then do the same with the newest Crafty you can find. I don't know which old Crafty's you can still get but even if it's not very old it would still be useful. The challenge will be to find a program that still runs on both platforms.

I know that is probably a bit of work - but if you are willing to do it I would like to see the results.

What about running a Bochs IA32 emulator on modern hardware? Maybe it can be hacked to run at some arbitrary virtual clock speed.

But we cannot agree on what that speed should be. Bob thinks 1000 to 1 and I think 100 to 1 (based on on actual test with Rebel.)

And it might be 100x with rebel. But I assume you saw my numbers from last night? P5/133 was 30K. P5/90 was in the 20K range. 1995 Crafty is slightly faster than 2010 crafty in NPS, and that is from a somewhat unfair comparison in that 2010 has full PGO optimization, which gains another 10%+ in speed, while the 1995 version did not. 1995 still has another 10% to gain once I rewrite the Makefile.

1000x is not unreasonable. If you take the real P5/90 number of 20K, and divide that into the real 2010 hardware number which is going to be in the range of 33M if you assume 10% PGO speedup, you actually will get 1500x, which is a pretty accurate number for Crafty. Same program today runs about 1500x faster than it did on the P5/90. Or if you use the somewhat rounded numbers, and use the P5/133 nps which I know to be accurate (the P90 is a good guess) you still get 1000x. For Crafty. Your rebel/genius numbers are poor benchmarks. 32 bit programs. Which means (a) no 64 bit math at all, and (b) no r8-r15 extra registers. Both lobotomize the new hardware. Crafty uses it all since the 1995 version was compiled on a 64 bit OS, it is based on bitboards, and the compiler knows about all the new hardware features. All I changed from the 1995 version was the assembly code, because it had to use 64 bit naming conventions rather than 32 bit (%rax rather than %eax, for example) to compile and run.

[bob]

Just curious, what was the price of that 1995 Cray T90?

The one we used in the last ACM event sold for $70m. Had a ton of SSD disk drives, striped (now called raid 0) to produce some disk drive performance one can only dream about without very deep pockets. We usually ran on machines in the manufacturing center. Most Cray customers required a 30-day burn in. The machine had to run 30 days, non-stop, no hardware or software failures, before it would leave the plant and be delivered. Most were quite happy to let us run on their boxes since we stressed everything to the max, beating on memory, all the CPUs, the shared semaphore registers, etc... And we found quite a few bugs for Cray doing this.

[Don]

If Bob is really interesting in measure ACTUAL hardware improvement we should be comparing the best parallel alpha system running in 1995 to the best chip we have running in 2010, the i7-980x which is a real product.

Bob, are you interested in a fair comparison or do you just want to compare the highly inferior Intel product to the very best chip we have today?

The alpha systems cost more than the Pentiums, but they were affordable (as high end systems go.)

It would be _very_ difficult to locate 1995 hardware. And if you venture that far out, the alpha sucks. The Cray T932 was king-of-the-hill in 1995. But you aren't going to find one laying around. The alpha is gone. Pentium 90 to i7 represents at least a homogeneous series of processors. I just happened to have data on the p5/133, and it is trivial to get numbers for the i7 since many have them.

I don't believe it makes sense to take a processor that is not available today, and which was discontinued many years ago, and include that. Yes the alpha was better than intel at the time. But the newer Intels are far superior. With Intel we can at least compare a chip that was available in 1995 and compare it to chips available today. And to be honest, hardly anybody cares about anything but the PC platform since that is what 99.999999999% of the people are using to play chess.

That is all well and good, but that's not what we are debating. I thought we were talking about HARDWARE advancement, not Intel marketing hype.

And they have not even seen the exotic hardware some of us used to use. All the rating lists use Intel (or AMD, same thing). There is better hardware available today than a single 6-core i7 as well. But very few programs run on clusters, and hardly anyone has access to the really high-end clusters from IBM and others.

BTW my first parallel search ran in 1978 at an ACM event using a dual-CPU univac box. I was not the first. Monty Newborn was using a group of data general machines connected by a channel-to-channel adapter even earlier.

As far as your alpha question goes, how can we use it? Development was discontinued 10+ years ago.

It doesn't matter how we would use it or even if we can use it. The issue isn't what can we test but how much has hardware advanced.

If you want to change the rules and make this something else, for instance, "commercial chess programs on Windows" then we have a different thing.

I'm having an issue with this because the P90 reference point is pretty weak compared to what was easily available but the i7-980x reference point is the best system you can buy right now. I have a suggestion for resolving this - which I will bring up in my response to your next post.

Where would it be today if we had a 21564, 21664, etc? No way to know. The alpha was not particularly superior to Intel at the time, if you exclude the obvious 64 bit advantage it had for bitboards. Bruce ran on a cryo-version at the 1997 WMCCC, and reported that for ferret, the only advantage was the nearly 800mhz clock speed for the nitrogen-cooled processor. I saw more than that, but then I was 64 bits and Bruce was not. The alpha was pretty good for floating point, which was its major market in the scientific workstation market. Didn't help chess at all in that regard.

Didn't you say earlier that it should not count if you have to change your program to take advantage of the hardware? So by your reasoning would it be fair to use a 32 bit program to benchmark a 64 bit system?

[bob]

If Bob is really interesting in measure ACTUAL hardware improvement we should be comparing the best parallel alpha system running in 1995 to the best chip we have running in 2010, the i7-980x which is a real product.

Bob, are you interested in a fair comparison or do you just want to compare the highly inferior Intel product to the very best chip we have today?

The alpha systems cost more than the Pentiums, but they were affordable (as high end systems go.)

It would be _very_ difficult to locate 1995 hardware. And if you venture that far out, the alpha sucks. The Cray T932 was king-of-the-hill in 1995. But you aren't going to find one laying around. The alpha is gone. Pentium 90 to i7 represents at least a homogeneous series of processors. I just happened to have data on the p5/133, and it is trivial to get numbers for the i7 since many have them.

I don't believe it makes sense to take a processor that is not available today, and which was discontinued many years ago, and include that. Yes the alpha was better than intel at the time. But the newer Intels are far superior. With Intel we can at least compare a chip that was available in 1995 and compare it to chips available today. And to be honest, hardly anybody cares about anything but the PC platform since that is what 99.999999999% of the people are using to play chess.

That is all well and good, but that's not what we are debating. I thought we were talking about HARDWARE advancement, not Intel marketing hype.

Then the debate is over before it starts. Cray T932 was best 1995 hardware platform, _period_. It is probably equal to any single chip computer today. So we have _zero_ hardware improvement. Is there anything left to discuss? Or we can take the most-commonly used hardware, which is clearly the PC, and compare that. Nobody was using alphas to play against humans in tournaments in 1995. They were using PCs. Nobody has used alphas to challenge GM players in any recent tournament. Nobody has used alphas in any comp vs comp tournament in the last 10 years. Nor have they used Crays.

So what do we use for the discussion?

And they have not even seen the exotic hardware some of us used to use. All the rating lists use Intel (or AMD, same thing). There is better hardware available today than a single 6-core i7 as well. But very few programs run on clusters, and hardly anyone has access to the really high-end clusters from IBM and others.

BTW my first parallel search ran in 1978 at an ACM event using a dual-CPU univac box. I was not the first. Monty Newborn was using a group of data general machines connected by a channel-to-channel adapter even earlier.

As far as your alpha question goes, how can we use it? Development was discontinued 10+ years ago.

It doesn't matter how we would use it or even if we can use it. The issue isn't what can we test but how much has hardware advanced.

That is simple enough. 1995 21064 was not a fast box. Unless you needed floating point and pretty good memory bandwidth (but not latency).

If you want to change the rules and make this something else, for instance, "commercial chess programs on Windows" then we have a different thing.

I'm having an issue with this because the P90 reference point is pretty weak compared to what was easily available but the i7-980x reference point is the best system you can buy right now. I have a suggestion for resolving this - which I will bring up in my response to your next post.

That last statement is not correct. One can find 4 and 8 chip i7's, which means 24 to 48 cores. And there are vendors going beyond that. So on one end you want to pick a single chip i7, and then want to quibble about it's "mate" from 1995? I've already shown that for Crafty, P90 to my 3+ year old stuff is almost 1500x. For crafty, using 1000x is not an "overstatement" at all. It is significantly lower than it should be.

But let's do a reality check because you are taking this into left field.

In 1995, what was 99.9% of chess engines running on? Whether playing in human events, computer events, mixed events, basement tournaments, used for game analysis, etc. P5/90 or slower Intel cpu.

In 2010, what is the best Intel processor around? I7. We have already cut the advantage by only allowing a parallel search on at most 6 cores (single chip) when duals and quads are everywhere (we have a dual -6core cluster on order right now, in fact).

Alpha is irrelevant unless we factor in Cray, TMI, Hitachi, Fujitsu, MIPS, Cray's newer distributed memory AMD boxes (thousands of cores) and so forth. What about the deep blue hardware? Faster than anything available today, and it was 1995-1997 era. One has to draw the line somewhere. I think most would say that the current discussion between P5 (1995) and i7(2010) is by far the most appropriate (and most interesting) idea. Otherwise no comparison is possible since we can't resurrect DB as the best there was in 1997, and there's only one software program that would run on that without a _ton_ of effort.

Where would it be today if we had a 21564, 21664, etc? No way to know. The alpha was not particularly superior to Intel at the time, if you exclude the obvious 64 bit advantage it had for bitboards. Bruce ran on a cryo-version at the 1997 WMCCC, and reported that for ferret, the only advantage was the nearly 800mhz clock speed for the nitrogen-cooled processor. I saw more than that, but then I was 64 bits and Bruce was not. The alpha was pretty good for floating point, which was its major market in the scientific workstation market. Didn't help chess at all in that regard.

Didn't you say earlier that it should not count if you have to change your program to take advantage of the hardware? So by your reasoning would it be fair to use a 32 bit program to benchmark a 64 bit system?

No, you are just attempting to twist what I said. We are talking hardware. To take advantage of some hardware advances, one _must_ make software changes. I do not count those "changes" as "software improvements" in this context, because the changes by themselves do nothing unless you have the new hardware the changes need.

I do not quite see the point for all the tangents. It would seem to me we have a pretty good idea of what/how to test. I have a 1995 version of Crafty that seems to be running correctly on my cluster, and seems to be running well over 1000x faster than it did on 1995 P90. I am getting close to being able to announce just how strong (or weak) that 1995 program is on today's hardware, We can subtract that from Crafty's rating, add whatever fudge factor is needed to raise the standard to Rybka, and voila' we have the software gain from 1995. Then we can either argue about whether 1000x - 1500x hardware will be as big, bigger, or smaller. And hopefully I can come up with a way to to make crafty (1995 crafty) run at 22K nodes per second to see what that does to the rating. Then we have both, actually measured...

At least for those interested in hardware advancements that actually matter to 99.9% of the people fooling around with computer chess.

[Don]

The alpha was not particularly superior to Intel at the time, if you exclude the obvious 64 bit advantage it had for bitboards. ...

I meant to comment on this particular quote. I wrote a program for MIT that was to work on both the Pentium and Alpha and it did run on both. It was immediately apparent that there was no comparison. I wish I had real numbers to report but it was a dog on the Pentium compared to the Alpha. The 64 bit issue gives you some, but it's not even a doubling. But I remember that if you assumed the 64 bit on 32 bit Pentium crippled it (even) by 2 to 1, we were something like 2 or 3 times slower. The alpha was ridiculously superior.

But what do you think? Should we base the superiority of a 64 bit chip on a program written for a 32 bit program? You have already given your answer so it's a rhetorical question.

By the way, I DO agree with your reasoning that hardware speed measurements should take into consideration the hardware the program was designed on. But it's VERY difficult to separate that from real program improvements. Any change should be solely in the interest of making the program compatible with new hardware, NOT an actual improvement to the program. This is for hardware comparisons only of course. A reasonable Litmus test for this would be to ask if that change could reasonably be applied to the old hardware.

However, I don't believe that is a huge issue. I don't think it's more than 3/2 but I would be generous and allow 2 to 1 for this. The single biggest thing you could do is recompile the old source code on modern hardware. That will probably make up the biggest difference because it's only fair when comparing hardware with software benchmarks that the software is compiled to the target machine. I agree with you on this one.