harware vs software advances

[mhull]

I think that the question is also what time control to use(even if we agree to take the new program and run it on old hardware).

I expect the new programs to get relatively better result at 120/40 time control relative to blitz.

Anpther problem is that some new programs may not be able to run on old hardware(for example if they use some tables that are too big for the old hardware to remember) and the question is what is the time of the old hardware.

I think that if we compare with year 2000 we are going to get at least 400 elo improvement only by software even if we use hardware of 2000 for every program and not very fast time control(let say at least 10 minutes for 40 moves).

hopefully there are people with some hardware of 2000 who can test it.

I am using ultra-short time controls quite often, something like games in 6 seconds. The new strong engines are beating the crap out of old strong engines (from year 2000 or so) on the same hardware even at this time control, using a hash of 4MB, available at that time. The difference of software in the last 10 years is _at_least_ 400 Elo points, the difference in hardware is ~350 Elo points. I don't really understand what the whole discussion is about, I think I once wrote that the software development is a little faster than the hardware one, on average 40-45 Elo points per year software, ~35 Elo points per year hardware.

Year 2000 hardware is already too fast for this comparison which is why we were talking about 1995 hardware. It seems to me (and I could be wrong) that at some point, the path length per node will become too long for the advanced algorithms to see their advantage on the slower hardware, and so on such hardware, the knowledge we have now could not be realized in software. Perhaps an Intel P90 is too slow. If not, maybe a 680x0. If that is not slow enough, then perhaps it's not really that interesting after all.

[Laskos]

Year 2000 hardware is already too fast for this comparison which is why we were talking about 1995 hardware. It seems to me (and I could be wrong) that at some point, the path length per node will become too long for the advanced algorithms to see their advantage on the slower hardware, and so on such hardware, the knowledge we have now could not be realized in software. Perhaps an Intel P90 is too slow. If not, maybe a 680x0. If that is not slow enough, then perhaps it's not really that interesting after all.

Here is my result with Houdini1.5 and Shredder 9 at 0.05 sec/move (~6 secs/game) and 1MB hash. Shredder 9 is ~20 Elo points stronger than Shredder 7.04 (according to SSDF), the best engine in 2002. I simply do not have an older UCI highest rated engine.

Code: Select all

    Program                            Score       %      Elo    +   -    Draws

  1 Houdini 1.5a x64               : 3067.5/3200  95.9    3273   26  25    4.8 %
  2 Deep Shredder 9                :  132.5/3200   4.1    2727   25  26    4.8 %

The difference is 546 +/- 25 Elo points (95% confidence). Thus, the year 2002 highest ranked Shredder 7.04 is ~566 +/- 25 Elo points weaker than Houdini 1.5 on the same hardware at 0.05 sec/move, a control hundreds of times shorter than a normal one, being pretty much equivalent to the slower, 12-14 years old hardware at normal TC.

This experiment shows that in 9 years the software improved by ~566 Elo points, while hardware by barely 300-350 Elo points. Or, ~60 Elo points per year software, ~35 Elo points per year hardware.

Kai

[mhull]

Year 2000 hardware is already too fast for this comparison which is why we were talking about 1995 hardware. It seems to me (and I could be wrong) that at some point, the path length per node will become too long for the advanced algorithms to see their advantage on the slower hardware, and so on such hardware, the knowledge we have now could not be realized in software. Perhaps an Intel P90 is too slow. If not, maybe a 680x0. If that is not slow enough, then perhaps it's not really that interesting after all.

Here is my result with Houdini1.5 and Shredder 9 at 0.05 sec/move (~6 secs/game) and 1MB hash. Shredder 9 is ~20 Elo points stronger than Shredder 7.04 (according to SSDF), the best engine in 2002. I simply do not have an older UCI highest rated engine.

Code: Select all

    Program                            Score       %      Elo    +   -    Draws

  1 Houdini 1.5a x64               : 3067.5/3200  95.9    3273   26  25    4.8 %
  2 Deep Shredder 9                :  132.5/3200   4.1    2727   25  26    4.8 %

The difference is 546 +/- 25 Elo points (95% confidence). Thus, the year 2002 highest ranked Shredder 7.04 is ~566 +/- 25 Elo points weaker than Houdini 1.5 on the same hardware at 0.05 sec/move, a control hundreds of times shorter than a normal one, being pretty much equivalent to the slower, 12-14 years old hardware at normal TC.

This experiment shows that in 9 years the software improved by ~566 Elo points, while hardware by barely 300-350 Elo points. Or, ~60 Elo points per year software, ~35 Elo points per year hardware.

Kai

That version of Shredder is not optimized for modern hardware which I would assume to be a handicap and not representative of the true hardware improvement. More confidence in this regard would require the finding of a true speedup for at least one of the programs being compared, preferably the old program. A strong open source program with available NPS from logs of the historical period would provide a hardware baseline. Then a version of the old program, re-optimized and tuned for modern hardware should be measured and its average NPS compared to the historical record. That would be the true hardware speed ratio which could then be applied to any program in the comparison by either time control or induced NPS reduction (delay loops within each program).

I would expect the differences seen your tests to be somewhat less, though by how much would only be a guess at this point.

[Laskos]

Year 2000 hardware is already too fast for this comparison which is why we were talking about 1995 hardware. It seems to me (and I could be wrong) that at some point, the path length per node will become too long for the advanced algorithms to see their advantage on the slower hardware, and so on such hardware, the knowledge we have now could not be realized in software. Perhaps an Intel P90 is too slow. If not, maybe a 680x0. If that is not slow enough, then perhaps it's not really that interesting after all.

Here is my result with Houdini1.5 and Shredder 9 at 0.05 sec/move (~6 secs/game) and 1MB hash. Shredder 9 is ~20 Elo points stronger than Shredder 7.04 (according to SSDF), the best engine in 2002. I simply do not have an older UCI highest rated engine.

Code: Select all

    Program                            Score       %      Elo    +   -    Draws

  1 Houdini 1.5a x64               : 3067.5/3200  95.9    3273   26  25    4.8 %
  2 Deep Shredder 9                :  132.5/3200   4.1    2727   25  26    4.8 %

The difference is 546 +/- 25 Elo points (95% confidence). Thus, the year 2002 highest ranked Shredder 7.04 is ~566 +/- 25 Elo points weaker than Houdini 1.5 on the same hardware at 0.05 sec/move, a control hundreds of times shorter than a normal one, being pretty much equivalent to the slower, 12-14 years old hardware at normal TC.

This experiment shows that in 9 years the software improved by ~566 Elo points, while hardware by barely 300-350 Elo points. Or, ~60 Elo points per year software, ~35 Elo points per year hardware.

Kai

That version of Shredder is not optimized for modern hardware which I would assume to be a handicap and not representative of the true hardware improvement. More confidence in this regard would require the finding of a true speedup for at least one of the programs being compared, preferably the old program. A strong open source program with available NPS from logs of the historical period would provide a hardware baseline. Then a version of the old program, re-optimized and tuned for modern hardware should be measured and its average NPS compared to the historical record. That would be the true hardware speed ratio which could then be applied to any program in the comparison by either time control or induced NPS reduction (delay loops within each program).

I would expect the differences seen your tests to be somewhat less, though by how much would only be a guess at this point.

I cannot take these arguments as valid. As I observed during the last 10 years, Fritz and Crafty are among the fastest in NPS disregarding the hardware, Shredder about two times slower and Hiarcs around 3 times slower. These things didn't changed a lot in these 10 years, and even an improbable variation of 50% would change 40 Elo points in 10 Years of software improvement, 4 points per year.

You could take like that
per year:
~60 +/- 10 Elo points software advance
~35 Elo points hardware advance

All averaged over last 10 years.

Now Bob will come to say that it's not true, and I will agree with him (in the case of Crafty ).

Kai

[mhull]

I cannot take these arguments as valid. As I observed during the last 10 years, Fritz and Crafty are among the fastest in NPS disregarding the hardware, Shredder about two times slower and Hiarcs around 3 times slower. These things didn't changed a lot in these 10 years, and even an improbable variation of 50% would change 40 Elo points in 10 Years of software improvement, 4 points per year.

But what if old shredder could run 50% faster on modern hardware if rewritten for modern hardware? What about 100% faster? Then your current estimate for old shredder on modern hardware is too low, and the software improvement estimate is too high.

[Laskos]

I cannot take these arguments as valid. As I observed during the last 10 years, Fritz and Crafty are among the fastest in NPS disregarding the hardware, Shredder about two times slower and Hiarcs around 3 times slower. These things didn't changed a lot in these 10 years, and even an improbable variation of 50% would change 40 Elo points in 10 Years of software improvement, 4 points per year.

But what if old shredder could run 50% faster on modern hardware if rewritten for modern hardware? What about 100% faster? Then your current estimate for old shredder on modern hardware is too low, and the software improvement estimate is too high.

100% faster means ~70 Elo points, and if that happened in 10 years, then it's 7 points per year. Some miracle should have happened to change my values dramatically, and I don't believe that even 50% faster is realistic, basically nothing important happened about the way of coding and optimization in the last 10 years.

Kai

[Vinvin]

You could take like that
per year:
~60 +/- 10 Elo points software advance
~35 Elo points hardware advance
Kai

I the case of the hardware, there was actually sudden jumps : introduction of first 2 cores computers, 4 cores computers ...

[Uri Blass]

I cannot take these arguments as valid. As I observed during the last 10 years, Fritz and Crafty are among the fastest in NPS disregarding the hardware, Shredder about two times slower and Hiarcs around 3 times slower. These things didn't changed a lot in these 10 years, and even an improbable variation of 50% would change 40 Elo points in 10 Years of software improvement, 4 points per year.

But what if old shredder could run 50% faster on modern hardware if rewritten for modern hardware? What about 100% faster? Then your current estimate for old shredder on modern hardware is too low, and the software improvement estimate is too high.

100% faster means ~70 Elo points, and if that happened in 10 years, then it's 7 points per year. Some miracle should have happened to change my values dramatically, and I don't believe that even 50% faster is realistic, basically nothing important happened about the way of coding and optimization in the last 10 years.

Kai

I think that one improvement is that we have better compilers so even with the same source code you can get some speed improvement.

I also do not believe that the speed improvement that you can get with the same code is very big and I also do not believe that even 50% faster is realistic but I also believe that with fast time control 100% faster means more than 70 elo improvement.

[rbarreira]

I cannot take these arguments as valid. As I observed during the last 10 years, Fritz and Crafty are among the fastest in NPS disregarding the hardware, Shredder about two times slower and Hiarcs around 3 times slower. These things didn't changed a lot in these 10 years, and even an improbable variation of 50% would change 40 Elo points in 10 Years of software improvement, 4 points per year.

But what if old shredder could run 50% faster on modern hardware if rewritten for modern hardware? What about 100% faster? Then your current estimate for old shredder on modern hardware is too low, and the software improvement estimate is too high.

100% faster means ~70 Elo points, and if that happened in 10 years, then it's 7 points per year. Some miracle should have happened to change my values dramatically, and I don't believe that even 50% faster is realistic, basically nothing important happened about the way of coding and optimization in the last 10 years.

Kai

I think that one improvement is that we have better compilers so even with the same source code you can get some speed improvement.

I also do not believe that the speed improvement that you can get with the same code is very big and I also do not believe that even 50% faster is realistic but I also believe that with fast time control 100% faster means more than 70 elo improvement.

gcc 4.5 generates much faster bitboard code than gcc 4.1 did for example. Using hardware popcnt can give another 10% boost, so I think 50% from better compilers and better use of modern hardware is definitely possible. In fact it should be more, if the transition from 32 bit to 64 bit is also counted.

[Laskos]

I cannot take these arguments as valid. As I observed during the last 10 years, Fritz and Crafty are among the fastest in NPS disregarding the hardware, Shredder about two times slower and Hiarcs around 3 times slower. These things didn't changed a lot in these 10 years, and even an improbable variation of 50% would change 40 Elo points in 10 Years of software improvement, 4 points per year.

But what if old shredder could run 50% faster on modern hardware if rewritten for modern hardware? What about 100% faster? Then your current estimate for old shredder on modern hardware is too low, and the software improvement estimate is too high.

100% faster means ~70 Elo points, and if that happened in 10 years, then it's 7 points per year. Some miracle should have happened to change my values dramatically, and I don't believe that even 50% faster is realistic, basically nothing important happened about the way of coding and optimization in the last 10 years.

Kai

I think that one improvement is that we have better compilers so even with the same source code you can get some speed improvement.

I also do not believe that the speed improvement that you can get with the same code is very big and I also do not believe that even 50% faster is realistic but I also believe that with fast time control 100% faster means more than 70 elo improvement.

gcc 4.5 generates much faster bitboard code than gcc 4.1 did for example. Using hardware popcnt can give another 10% boost, so I think 50% from better compilers and better use of modern hardware is definitely possible. In fact it should be more, if the transition from 32 bit to 64 bit is also counted.

Say a total of 80% boost in 10 years. I can agree with Uri that doubling at that short time control means more than 70 Elo points, but not by much, my estimation is 85 Elo points. Then a boost of 80% in speed over the last 10 years would give ~70 Elo points. Anyway, _averaged_ (to correct me according to Vincent) the per year improvements are:

55 +/- 10 Elo points (95%) software
~35 Elo points hardware

on average over the last 10 years using normal time controls.

Kai