Nakamura vs Stockfish, public match 8/23

[Milos]

I have to agree with you on this. The past 23 years the hardware speed increased 1000 fold, this accounts for something like 700 ELO.
I still remember running on one of the 'super computers' from our national Dutch computer center 'Sara' in 1991 where my program did about 15knps.
Nowadays it does 15mnps on a fast home computer.
I also believe that the increased hardware speed made some software tricks possible that were not feasible on slow hardware.
It is my estimation that the improvement due to software is something like 350 ELO which is not bad either.

Hardware improved nowhere near 1000 fold. There are threads here where this was discussed in detail, and different ppl did a lot of tests.
Basically, in last 20 years hardware sped up 150-200x which is around 500Elo, while in software you got 700-1000Elo (depending on which same hardware you compare old and new software - on old one, new one, or something intermediate).
Software improvements are still more important than hardware advance.

I,ve been in computer chess for 38 years and I have used and owned almost every hardware in existence, I remember very clearly that my program was running at 7 knps on a fast PC in 1991. Now it is running at 2000 times that speed. Maybe this is partly due to better compilers but 1000x is certainly possible.

And you used the same eval function (same pawn hash, same bitboards, same asm code for bsf/bsr), same move generator in 1991. and today? Yea right

Of course not, nowadays you can take advantage of the much better hardware like I said.

What you say is just comparing apples and oranges, you compare your old inferior software nps with newest software nps and attribute speed difference to hardware only. Excuse me, but that are just bullocks.
If you want to compare hardware only, you have to run exactly the same software code compiled specifically for old and new hardware.
Optimizing (speeding up) software for the specific hardware is still software part, not hardware part.

[Joost Buijs]

I have to agree with you on this. The past 23 years the hardware speed increased 1000 fold, this accounts for something like 700 ELO.
I still remember running on one of the 'super computers' from our national Dutch computer center 'Sara' in 1991 where my program did about 15knps.
Nowadays it does 15mnps on a fast home computer.
I also believe that the increased hardware speed made some software tricks possible that were not feasible on slow hardware.
It is my estimation that the improvement due to software is something like 350 ELO which is not bad either.

Hardware improved nowhere near 1000 fold. There are threads here where this was discussed in detail, and different ppl did a lot of tests.
Basically, in last 20 years hardware sped up 150-200x which is around 500Elo, while in software you got 700-1000Elo (depending on which same hardware you compare old and new software - on old one, new one, or something intermediate).
Software improvements are still more important than hardware advance.

I,ve been in computer chess for 38 years and I have used and owned almost every hardware in existence, I remember very clearly that my program was running at 7 knps on a fast PC in 1991. Now it is running at 2000 times that speed. Maybe this is partly due to better compilers but 1000x is certainly possible.

And you used the same eval function (same pawn hash, same bitboards, same asm code for bsf/bsr), same move generator in 1991. and today? Yea right

Of course not, nowadays you can take advantage of the much better hardware like I said.

What you say is just comparing apples and oranges, you compare your old inferior software nps with newest software nps and attribute speed difference to hardware only. Excuse me, but that are just bullocks.
If you want to compare hardware only, you have to run exactly the same software code compiled specifically for old and new hardware.
Optimizing (speeding up) software for the specific hardware is still software part, not hardware part.

It is just how you look at it.
When I'm able to use e.g. POPCOUNT in hardware instead of having to calculate it in software this is a virtue of the hardware and not of the software, the same holds for the use of 64 bit vs 32 or 16 bit and use of the larger and faster memory.

Anyway this discussion can go on for ever and we will never agree.

[Laskos]

Anyway this discussion can go on for ever and we will never agree.

No, it's not a matter of which colour one prefers. It's a matter that you claimed that 600 Elo points gain from software is less than 250 Elo points gain from hardware during the past 10 years. There is 350 Elo points difference in our claims. That is, 5 doublings at 40/4', or a factor of 32, or 3200% speed-up. You claim that things like POPCOUNT, which give a speed-up of less than 10%, are making up for those 3200%. Your claim is simply wrong, and while everyone is free to have opinions, I would suggest to keep your opinions for yourself, and not dis-inform people on a forum.

So, as a conclusion, in the last 10 years software improvement is about 600 Elo points, hardware about 250 Elo points.

[Sedat Canbaz]

Anyway this discussion can go on for ever and we will never agree.

No, it's not a matter of which colour one prefers. It's a matter that you claimed that 600 Elo points gain from software is less than 250 Elo points gain from hardware during the past 10 years. There is 350 Elo points difference in our claims. That is, 5 doublings at 40/4', or a factor of 32, or 3200% speed-up. You claim that things like POPCOUNT, which give a speed-up of less than 10%, are making up for those 3200%. Your claim is simply wrong, and while everyone is free to have opinions, I would suggest to keep your opinions for yourself, and not dis-inform people on a forum.

So, as a conclusion, in the last 10 years software improvement is about 600 Elo points, hardware about 250 Elo points.

+1

Just I'd like to add,

10 years ago, the top engines were: Shredder 8, Hiarcs 9, Fritz 8 etc...
As decent hardwares were: AMD Athlon 64 3400+ 2.40GHz

For example, in those days I had a rating list:
http://www.stmintz.com/ccc/index.php?id=396038

And all the above mentioned old top engines were on near level as many Top GMs (in 2004 year)

So, looking at CCRL (40/4)

Fritz 8 is rated around 2667 Elo
Stockfish 5 is rated around 3242 Elo

Engine Elo difference is approx. 575 Elo

And now let's calcvulate the Stockfish bench hardware speed:
2x Xeon E5-2697 v2 @3.22GHz 23020 kns
AMD Athlon 64 3400+ 2.40GHz 700 kns

700 kns = 0 Elo
1400 kns = 75 Elo
2800 kns = 150 Elo
5600 kns = 225 Elo
11200 kns = 300 Elo
22400 kns = 375 Elo

Hardware Elo difference is approx. 375 Elo

Note also that since 2004 and 2014: the opening theory has been changed a lot

So, here are the overall improvements since 2004 up to 2014:
Engine Elo Improvement: 575 Elo
Hardware Speed Improvement: 375 Elo
Opening Book Improvement at least: 100 Elo
Total Improvement: 1050 Elo

In other words,
Stockfish 5 x64 24c (on 2x Xeon E5-2697 v2) shoud be rated at least 3600 Elo !!

And it is a pitty that many people still can not see the reality...!
For example, some testers give very low engine Elo values...they need to increase their Elo start calculations...

Best,
Sedat

[Joost Buijs]

Anyway this discussion can go on for ever and we will never agree.

No, it's not a matter of which colour one prefers. It's a matter that you claimed that 600 Elo points gain from software is less than 250 Elo points gain from hardware during the past 10 years. There is 350 Elo points difference in our claims. That is, 5 doublings at 40/4', or a factor of 32, or 3200% speed-up. You claim that things like POPCOUNT, which give a speed-up of less than 10%, are making up for those 3200%. Your claim is simply wrong, and while everyone is free to have opinions, I would suggest to keep your opinions for yourself, and not dis-inform people on a forum.

So, as a conclusion, in the last 10 years software improvement is about 600 Elo points, hardware about 250 Elo points.

I said software improvement <= 350 Elo for the last ten years, I never spoke about < 250 Elo anywhere.

And I like to vent my opinions whenever I want, I don't need any suggestions from you in this respect.

[Uri Blass]

The titled-player commentators are calling out computer engine programmers saying that they have taken too much credit in their creations' strength over humans when it's mostly been the inexorable progress of hardware speed.

That's simply false, but what do those guys know about chess...

I suspect that there is more right about their statement than wrong. That is, I believe that well beyond 50% of the improvements over the past 20 years has been hardware related. I REALLY want to take my old Cray Blitz source, circa 1989, and run it on today's hardware and test it on my cluster to see where it comes in. It had no reductions or forward pruning other than null-move R=1. Only work I need to do is add basic xboard protocol support as it was not xboard compatible. But it used the same "seaboard" command as Crafty so that is done, and it inputs and outputs SAN moves. All I really need is time/otim and it should probably work on my cluster.

I remember running Cray Blitz (fortran-only code of course) on a Pentium 133mhz box and hit about 100 nodes per second. I got this old version to compile cleanly a couple of years ago but don't remember the speed on more modern hardware. 64 bit hardware should be even better...

Be an interesting test.

I have to agree with you on this. The past 23 years the hardware speed increased 1000 fold, this accounts for something like 700 ELO.
I still remember running on one of the 'super computers' from our national Dutch computer center 'Sara' in 1991 where my program did about 15knps.
Nowadays it does 15mnps on a fast home computer.
I also believe that the increased hardware speed made some software tricks possible that were not feasible on slow hardware.
It is my estimation that the improvement due to software is something like 350 ELO which is not bad either.

Your estimation for the improvement due to software is clearly wrong.

Software improved clearly more than 350 elo and you do not get results of 99:1 by 350 elo improvement espacially when the old programs get more time on the same hardware in ponder off games.

We also talk about the last 20 years and not about the last 23 years.

I believe that there were some years when most of the improvement were due to hardware but it stopped in 2004 when fabien released fruit and if you look at the progress in software in the last 10 years then you will find that it is clearly bigger than the improvement in hardware in the last 10 years.

It is possible that the last 10 years the improvement in software is bigger then the one in hardware but I doubt it.
Anyway the total progress made in software the last 23 years is about 350 ELO so for the last 10 years it will always be equal or less.

Not in comp-comp games and the software improvement based on comp-comp games is bigger than 350 elo.
99-1 clearly suggest bigger difference than 350 elo and the winner got 5 minutes against 15 minutes of the loser in every game.

[syzygy]

I believe that there were some years when most of the improvement were due to hardware but it stopped in 2004 when fabien released fruit and if you look at the progress in software in the last 10 years then you will find that it is clearly bigger than the improvement in hardware in the last 10 years.

Indeed. Just take Fruit 2.1 which must have been released somewhere in 2005 and at that time was at least close to the top. On CCRL 40/40 it has a rating of 2693. SF5 single cpu has a rating of 3212. That is more than 500 Elo software improvement in about 9 years.

[Sedat Canbaz]

I believe that there were some years when most of the improvement were due to hardware but it stopped in 2004 when fabien released fruit and if you look at the progress in software in the last 10 years then you will find that it is clearly bigger than the improvement in hardware in the last 10 years.

Indeed. Just take Fruit 2.1 which must have been released somewhere in 2005 and at that time was at least close to the top. On CCRL 40/40 it has a rating of 2693. SF5 single cpu has a rating of 3212. That is more than 500 Elo software improvement in about 9 years.

Yes... but as far as I know,
Uri disagrees that Stockfish 5 should be rated around 3500 Elo (on decent hardwares )))

His estimation is around 3200 Elo ))

[syzygy]

I believe that there were some years when most of the improvement were due to hardware but it stopped in 2004 when fabien released fruit and if you look at the progress in software in the last 10 years then you will find that it is clearly bigger than the improvement in hardware in the last 10 years.

Indeed. Just take Fruit 2.1 which must have been released somewhere in 2005 and at that time was at least close to the top. On CCRL 40/40 it has a rating of 2693. SF5 single cpu has a rating of 3212. That is more than 500 Elo software improvement in about 9 years.

Yes... but as far as I know,
Uri disagrees that Stockfish 5 should be rated around 3500 Elo (on decent hardwares )))

His estimation is around 3200 Elo ))

I agree with your comparison between SF5 and Fritz 8 on CCRL, but I think 375 Elo is too high for 5 speed doublings which includes going from 1 to 24 threads. 250 Elo might be more accurate.

And of course at least part of the Elo gained by multithreaded search should be attributed to software. It's not exactly trivial to make an engine run on many cores.

[Sedat Canbaz]

I agree with your comparison between SF5 and Fritz 8 on CCRL, but I think 375 Elo is too high for 5 speed doublings which includes going from 1 to 24 threads. 250 Elo might be more accurate.

And of course at least part of the Elo gained by multithreaded search should be attributed to software. It's not exactly trivial to make an engine run on many cores.

Hello Ronald )

Who said that since 2004 and 2014: the hardware speed difference should be 250 Elo ??

I can give you another examples, based on my experience and in Auto232 mode matches !!
https://sites.google.com/site/computers ... ct-auto232

And I hope the below results will be helpful for comparing:

Code: Select all

Engine                        Elo      Hardware Speed      Stockfish Bench
Deep Rybka 4.1 x64 6c        3358 Elo  i7 980X 4.33GHz       10512 kN/s 
Deep Rybka 4.1 x64 4c        3293 Elo  Intel Core i7 920      5229 kN/s 
Rybka 4.1 x64 1c             3199 Elo  QX9650  3.66 GHz       1380 kN/s

What about if we run (under below conditions) then I expect to see around:

Code: Select all

Deep Rybka 4.1 x64 24c        3458 Elo  2x Xeon E5-2697 v2    23020 kN/s 
Rybka 4.1 w32 1c              3100 Elo  AMD Athlon 64 3400+     700 kN/s

As we see the hardware Elo difference between 2004 and 2014 is approx. 360 Elo
I've checked rest engines too, for example Houdini 2.0c shows approx. 370 Elo

Correct me if I am wrong ?)

And if it is not enough the current my published data...I have another Hardware Elo tournament...just let me know.. ))

Btw, please don't look what all people will say... mostly of them have no any single game to prove...


Best,
Sedat