I would like to produce a graph showing the progress of the top chess computer over time - in one version as absolute numbers (elo vs year), and in another version, pure algorithmic progress (elo vs year of release, on identical hardware). I can find a lot of information on the web - for example, the CCRL rating list has lots of current and older programs running on equal hardware to compare. What I'm lacking though is a list of which chess program was best in a given year (to search the top program per year in the CCRL list).
Has anyone got an idea where I could find this info, or maybe already compiled such a list of progress in computer chess? I would especially like to be able to compare the influence that better hardware had and the influence that better software had.
One example of what I would like to do: using the CCRL (40/4) list but maybe there is some better resource?
2016: best engine is Stockfish with a single-CPU rating of 3246
2005: Fruit 2.1 was one of the best engines around, with a rating of 2693
2003: Ruffian 1.05 (?) was one of the best engines , with a rating of 2608
I would like to extend this list with more years, and further back, but I'm unsure of which engines were top when - can anyone help?
I find it very interesting that pure algorithmic progress between 2003 and 2016 yielded 600 rating points! Guesstimating 60 Elo for a speed doubling, 18 Months as doubling time from Moore's law, and 13 years, I get ~500 elo for hardware improvement in that time, so software appears to have made more progress than hardware. I would also be interested to hear your thoughts on what makes the difference between Stockfish and e.g. Ruffian - what was invented algorithm-wise in the last 10 years (LMR? what else?) and how much did it contribute to the improvement? Or is it all due to better testing?
best regards
Martin
I don't think you can accurately make the claim that software made more progress. There are some things we do in software today solely because the hardware is so fast we don't get penalized as much as we would have 10 or 20 years ago. So clock doubling is but a part of the hardware equation. This phenomenon has been observed for 50 years now, in fact. Prior to chess 4.x, nobody could get away with exhaustive search, it was way too expensive and the 2-3 ply depths were not enough to beat even barely decent humans. Until the Cyber 176 came along. We've seen that happen several times now.
Note that the question is about the time between 2003 and 2016 and not about what happened 20 or 30 years ago.
Chess programs of today cannot run on hardware that is 30 years old but still can use hardware of 2003(possibly with some small modifications).
I also expect Komodo with hardware of 2003 to beat every chess program of 2003 with hardware that we have today at least in every time control that is not blitz.
I would like to produce a graph showing the progress of the top chess computer over time - in one version as absolute numbers (elo vs year), and in another version, pure algorithmic progress (elo vs year of release, on identical hardware). I can find a lot of information on the web - for example, the CCRL rating list has lots of current and older programs running on equal hardware to compare. What I'm lacking though is a list of which chess program was best in a given year (to search the top program per year in the CCRL list).
Has anyone got an idea where I could find this info, or maybe already compiled such a list of progress in computer chess? I would especially like to be able to compare the influence that better hardware had and the influence that better software had.
One example of what I would like to do: using the CCRL (40/4) list but maybe there is some better resource?
2016: best engine is Stockfish with a single-CPU rating of 3246
2005: Fruit 2.1 was one of the best engines around, with a rating of 2693
2003: Ruffian 1.05 (?) was one of the best engines , with a rating of 2608
I would like to extend this list with more years, and further back, but I'm unsure of which engines were top when - can anyone help?
I find it very interesting that pure algorithmic progress between 2003 and 2016 yielded 600 rating points! Guesstimating 60 Elo for a speed doubling, 18 Months as doubling time from Moore's law, and 13 years, I get ~500 elo for hardware improvement in that time, so software appears to have made more progress than hardware. I would also be interested to hear your thoughts on what makes the difference between Stockfish and e.g. Ruffian - what was invented algorithm-wise in the last 10 years (LMR? what else?) and how much did it contribute to the improvement? Or is it all due to better testing?
best regards
Martin
I don't think you can accurately make the claim that software made more progress. There are some things we do in software today solely because the hardware is so fast we don't get penalized as much as we would have 10 or 20 years ago. So clock doubling is but a part of the hardware equation. This phenomenon has been observed for 50 years now, in fact. Prior to chess 4.x, nobody could get away with exhaustive search, it was way too expensive and the 2-3 ply depths were not enough to beat even barely decent humans. Until the Cyber 176 came along. We've seen that happen several times now.
Note that the question is about the time between 2003 and 2016 and not about what happened 20 or 30 years ago.
Chess programs of today cannot run on hardware that is 30 years old but still can use hardware of 2003(possibly with some small modifications).
I also expect Komodo with hardware of 2003 to beat every chess program of 2003 with hardware that we have today at least in every time control that is not blitz.
Uri
What are you talking about? A program of today can run on a Pentium pro from 1996 with no problems whatsoever. They can run on the original pentium with absolutely no problems either. 64 bit ints were supported back then with zero problems, I was using 'em.
I would like to produce a graph showing the progress of the top chess computer over time - in one version as absolute numbers (elo vs year), and in another version, pure algorithmic progress (elo vs year of release, on identical hardware). I can find a lot of information on the web - for example, the CCRL rating list has lots of current and older programs running on equal hardware to compare. What I'm lacking though is a list of which chess program was best in a given year (to search the top program per year in the CCRL list).
Has anyone got an idea where I could find this info, or maybe already compiled such a list of progress in computer chess? I would especially like to be able to compare the influence that better hardware had and the influence that better software had.
One example of what I would like to do: using the CCRL (40/4) list but maybe there is some better resource?
2016: best engine is Stockfish with a single-CPU rating of 3246
2005: Fruit 2.1 was one of the best engines around, with a rating of 2693
2003: Ruffian 1.05 (?) was one of the best engines , with a rating of 2608
I would like to extend this list with more years, and further back, but I'm unsure of which engines were top when - can anyone help?
I find it very interesting that pure algorithmic progress between 2003 and 2016 yielded 600 rating points! Guesstimating 60 Elo for a speed doubling, 18 Months as doubling time from Moore's law, and 13 years, I get ~500 elo for hardware improvement in that time, so software appears to have made more progress than hardware. I would also be interested to hear your thoughts on what makes the difference between Stockfish and e.g. Ruffian - what was invented algorithm-wise in the last 10 years (LMR? what else?) and how much did it contribute to the improvement? Or is it all due to better testing?
best regards
Martin
I don't think you can accurately make the claim that software made more progress. There are some things we do in software today solely because the hardware is so fast we don't get penalized as much as we would have 10 or 20 years ago. So clock doubling is but a part of the hardware equation. This phenomenon has been observed for 50 years now, in fact. Prior to chess 4.x, nobody could get away with exhaustive search, it was way too expensive and the 2-3 ply depths were not enough to beat even barely decent humans. Until the Cyber 176 came along. We've seen that happen several times now.
I would be very surprised if most of the gain over the last say 15 years was not due to better software. There is a ton of evidence for this, but if someone has a suitable "high end" machine from 15 years ago, I could compile Komodo for it and have it run against the best software of its day. It would be an interesting experiment. I predict the highly selective search schemes we use now will beat any of the old programs from 15 years ago. Granted there are differences in memory but PCs in 2000 had 128 MB or memory, running at say 600 MHz. Eventually if you go far enough in the past there will not be enough memory to do some of the things we do now, but I think a stripped down version of Komodo would do well. I suspect this since it runs well on a cheap Android phone without much memory or hig processor speed. Albert Silver even showed that the strongest program from years ago running on today;s hardware gets beat by Komodo running on a cell phone getting 1/50th the number of nodes per seconds as on the PC:
Frankly, hardware speeds are not increasing much lately, like they did in the past. So I think most gains are due to better software. Anyone willing to run the experiment, let me know.
An easier test would be to take an old program vs Komodo, and throttle them so that they can only search 50K nodes per second, which was a reasonable speed on the pentium pro 200 box. And then tune Komodo to play better. I'd bet you would find that the forward pruning needs to be toned down when you are only searching 50K nodes per second. And that things like mobility are likely to be computationally too expensive unless it is a very simple implementation.
I didn't say all the gain was from hardware, but I did say a LOT of the gains were enabled by better hardware. I can go back to the early days to recall that what worked at 5000 nodes per second did not come anywhere near working at 1 NPS.
Edit: Note that the 50K number was representative of Crafty on a pentium pro 200. No idea how my NPS compares to yours on equal hardware (obviously a non-parallel search to boot).
Recently I have tried running Rodent II at 32768 nodes per second. It was enough to get a blitz score of 52% against good old Comet B68, an engine from pre-LMR times, running 10-11 times as fast. What is interesting, Comet was much better in endgame, but struggled in the middlegame.
PK wrote:Recently I have tried running Rodent II at 32768 nodes per second. It was enough to get a blitz score of 52% against good old Comet B68, an engine from pre-LMR times, running 10-11 times as fast. What is interesting, Comet was much better in endgame, but struggled in the middlegame.
I would bet some experimentation with tuning Rodent would improve those results. What one can get away with at 3M nodes per second is quite a bit different from what you can get away with 100x (or 1000x) slower...
PK wrote:Recently I have tried running Rodent II at 32768 nodes per second. It was enough to get a blitz score of 52% against good old Comet B68, an engine from pre-LMR times, running 10-11 times as fast. What is interesting, Comet was much better in endgame, but struggled in the middlegame.
I would bet some experimentation with tuning Rodent would improve those results. What one can get away with at 3M nodes per second is quite a bit different from what you can get away with 100x (or 1000x) slower...
Most engines are already being tuned at ultra-bullet time controls.
PK wrote:Recently I have tried running Rodent II at 32768 nodes per second. It was enough to get a blitz score of 52% against good old Comet B68, an engine from pre-LMR times, running 10-11 times as fast. What is interesting, Comet was much better in endgame, but struggled in the middlegame.
I would bet some experimentation with tuning Rodent would improve those results. What one can get away with at 3M nodes per second is quite a bit different from what you can get away with 100x (or 1000x) slower...
Most engines are already being tuned at ultra-bullet time controls.
10s + 0.1s is not quite the same as 30K nodes per second...
Crafty is doing about 6M NPS for testing. at 10s + 0.1s, that turns into maybe about 10s / 30 + 0.1s per move, or almost 1/2 second. 3M nodes searched. A program searching 30K NPS would search only 15K nodes (or a little less)...
I would like to produce a graph showing the progress of the top chess computer over time - in one version as absolute numbers (elo vs year), and in another version, pure algorithmic progress (elo vs year of release, on identical hardware). I can find a lot of information on the web - for example, the CCRL rating list has lots of current and older programs running on equal hardware to compare. What I'm lacking though is a list of which chess program was best in a given year (to search the top program per year in the CCRL list).
Has anyone got an idea where I could find this info, or maybe already compiled such a list of progress in computer chess? I would especially like to be able to compare the influence that better hardware had and the influence that better software had.
One example of what I would like to do: using the CCRL (40/4) list but maybe there is some better resource?
2016: best engine is Stockfish with a single-CPU rating of 3246
2005: Fruit 2.1 was one of the best engines around, with a rating of 2693
2003: Ruffian 1.05 (?) was one of the best engines , with a rating of 2608
I would like to extend this list with more years, and further back, but I'm unsure of which engines were top when - can anyone help?
I find it very interesting that pure algorithmic progress between 2003 and 2016 yielded 600 rating points! Guesstimating 60 Elo for a speed doubling, 18 Months as doubling time from Moore's law, and 13 years, I get ~500 elo for hardware improvement in that time, so software appears to have made more progress than hardware. I would also be interested to hear your thoughts on what makes the difference between Stockfish and e.g. Ruffian - what was invented algorithm-wise in the last 10 years (LMR? what else?) and how much did it contribute to the improvement? Or is it all due to better testing?
best regards
Martin
I don't think you can accurately make the claim that software made more progress. There are some things we do in software today solely because the hardware is so fast we don't get penalized as much as we would have 10 or 20 years ago. So clock doubling is but a part of the hardware equation. This phenomenon has been observed for 50 years now, in fact. Prior to chess 4.x, nobody could get away with exhaustive search, it was way too expensive and the 2-3 ply depths were not enough to beat even barely decent humans. Until the Cyber 176 came along. We've seen that happen several times now.
I would be very surprised if most of the gain over the last say 15 years was not due to better software. There is a ton of evidence for this, but if someone has a suitable "high end" machine from 15 years ago, I could compile Komodo for it and have it run against the best software of its day. It would be an interesting experiment. I predict the highly selective search schemes we use now will beat any of the old programs from 15 years ago. Granted there are differences in memory but PCs in 2000 had 128 MB or memory, running at say 600 MHz. Eventually if you go far enough in the past there will not be enough memory to do some of the things we do now, but I think a stripped down version of Komodo would do well. I suspect this since it runs well on a cheap Android phone without much memory or hig processor speed. Albert Silver even showed that the strongest program from years ago running on today;s hardware gets beat by Komodo running on a cell phone getting 1/50th the number of nodes per seconds as on the PC:
Frankly, hardware speeds are not increasing much lately, like they did in the past. So I think most gains are due to better software. Anyone willing to run the experiment, let me know.
I have a desktop which I bought back in 2003 that is laying around. It has an Intel pentium 4 running at 3.0 ghz. Unfortunately that is a single core processor but unless somebody has something more suitable it might do the trick. Other specs of the machine are that it is running windows xp and has 1 gig of ddr2 memory. I am not sure of the various bus speed and cache sizes as I am currently out on vaction. If that seems suitable I could run a match against Stockfish (other some other top engine) against zappa 1.1, which was one of the strongest engines in 2005.
PK wrote:Recently I have tried running Rodent II at 32768 nodes per second. It was enough to get a blitz score of 52% against good old Comet B68, an engine from pre-LMR times, running 10-11 times as fast. What is interesting, Comet was much better in endgame, but struggled in the middlegame.
I would bet some experimentation with tuning Rodent would improve those results. What one can get away with at 3M nodes per second is quite a bit different from what you can get away with 100x (or 1000x) slower...
Most engines are already being tuned at ultra-bullet time controls.
10s + 0.1s is not quite the same as 30K nodes per second...
Crafty is doing about 6M NPS for testing. at 10s + 0.1s, that turns into maybe about 10s / 30 + 0.1s per move, or almost 1/2 second. 3M nodes searched. A program searching 30K NPS would search only 15K nodes (or a little less)...
With 30K nodes per second you can get 3M nodes per move if you use 100 seconds per move.
I do not think that this type of time control was irrelevant at the time programs searched 30K nodes per second and the average time that you had in computer world championship is slower than 100 seconds per move.
PK wrote:Recently I have tried running Rodent II at 32768 nodes per second. It was enough to get a blitz score of 52% against good old Comet B68, an engine from pre-LMR times, running 10-11 times as fast. What is interesting, Comet was much better in endgame, but struggled in the middlegame.
I would bet some experimentation with tuning Rodent would improve those results. What one can get away with at 3M nodes per second is quite a bit different from what you can get away with 100x (or 1000x) slower...
Most engines are already being tuned at ultra-bullet time controls.
10s + 0.1s is not quite the same as 30K nodes per second...
Duh?
My obvious point is that nps does not determine the quality of the moves. The quality of the moves is fully determined by the number of moves searched. Ultra-bullet moves at high nps will be comparable to regular tc moves at 30Knps.