Dear all,
I have tried to add an experimental data point to the discussion on computer chess progress. Originally, I wanted to use Rebel Century or Rebel 6, but it turned out they didn't really adhere to Arena's time limit
So I ended up running Ruffian 1.05 (2002 if I am not mistaken, not the very best engine around then, but in the top group) against Stockfish 7 32-bit. I did this on an older Laptop (Pentium M, 1.5GHz, Win XP), 1000 games (starting from Ed Schröders work1.pgn) at 1s/move.
Ruffian was getting a bit more than 1MN/s, SF about half. Ruffian is 14 years old (so not quite the 20 years I had in mind, but close) and in the CCRL 40/40 list, stockfish 7 64 bit is on 3246, ruffian on 2608, a 638 point rating difference. SF 2.2.2 is on the list with both 32- and 64 bit versions, and has a 26 point rating difference there, so I could estimate a 612 point rating difference between SF7 and ruffian 1.0.5 according to the CCRL 40/40 list.
I would put 1s/move on the Pentium M 1.5GHz to around 100 times (s)lower than the CCRL 40/40 processing power. Theoretically it's possible that at lower NPS some things don't work any more. It's also possible that the rating list is sort of inconsistent because top engines are not playing older engines any more.
My experiment says otherwise: Stockfish won 974-26, Arena calculates a rating difference of 604 elo, really surprisingly close to the 612 of the CCRL list.
So I think I'll claim that this backs up my original claim of 750 elo over 20 years due to software alone, and that this would theoretically have been possible 20 years ago too - the algorithms just were not there. It would be nice to try even older top engines (Shredder or Fritz 5.32) but I have no access to them.
cheers
Martin
Computer Chess Progress: Stockfish 7 vs Ruffian 1.0.5
Moderators: hgm, Rebel, chrisw
-
- Posts: 72
- Joined: Mon Mar 07, 2016 4:41 pm
- Location: Zürich, Switzerland
-
- Posts: 710
- Joined: Sat Dec 06, 2014 1:53 pm
Re: Computer Chess Progress: Stockfish 7 vs Ruffian 1.0.5
Intel CPU history:
04.01.1996 - Pentium 166
10.06.1996 - Pentium 200
07.05.1997 - Pentium II 300
18.08.1997 - Pentium Pro 200
15.04.1998 - Pentium II 400
26.02.1999 - Pentium III 500
17.03.1999 - Xeon III 550
02.08.1999 - Pentium III 600
12.01.2000 - Pentium III 800
08.03.2000 - Pentium III 1000
20.11.2000 - Pentium 4 1500
23.04.2001 - Pentium 4 1700
02.06.2001 - Pentium 4 1800
27.08.2001 - Pentium 4 2000
07.01.2002 - Pentium 4 2200
02.04.2002 - Pentium 4 2400
26.08.2002 - Pentium 4 2600, FSB400
14.11.2002 - Pentium 4 3060, HT
12.03.2003 - Pentium M 1500 Centrino
23.09.2003 - Pentium 4 3200, FSB533
You run SF7 at pretty modern CPU (1M L2 cache, FSB400, SSE2)
Techniques which work well on 500-1000 kN/move might not work at 10-30 kn/move
P.S. I have alive PIII-500 (512 SDRAM, Windows XP) and could run some tests if there is interest
04.01.1996 - Pentium 166
10.06.1996 - Pentium 200
07.05.1997 - Pentium II 300
18.08.1997 - Pentium Pro 200
15.04.1998 - Pentium II 400
26.02.1999 - Pentium III 500
17.03.1999 - Xeon III 550
02.08.1999 - Pentium III 600
12.01.2000 - Pentium III 800
08.03.2000 - Pentium III 1000
20.11.2000 - Pentium 4 1500
23.04.2001 - Pentium 4 1700
02.06.2001 - Pentium 4 1800
27.08.2001 - Pentium 4 2000
07.01.2002 - Pentium 4 2200
02.04.2002 - Pentium 4 2400
26.08.2002 - Pentium 4 2600, FSB400
14.11.2002 - Pentium 4 3060, HT
12.03.2003 - Pentium M 1500 Centrino
23.09.2003 - Pentium 4 3200, FSB533
You run SF7 at pretty modern CPU (1M L2 cache, FSB400, SSE2)
Techniques which work well on 500-1000 kN/move might not work at 10-30 kn/move
P.S. I have alive PIII-500 (512 SDRAM, Windows XP) and could run some tests if there is interest
-
- Posts: 710
- Joined: Sat Dec 06, 2014 1:53 pm
Re: Computer Chess Progress: Stockfish 7 vs Ruffian 1.0.5
Hypo 1: At old CPU modern engine might slowdown much more than old engine.I'll claim that this backs up my original claim of 750 elo over 20 years due to software alone
If Ruffian is 2x faster than SF7 at modern Centrino CPU, it might be 5x faster at P-I
Hypo 2: Search techniqes which gain ELO at 100+ knodes per move might decrease ELO at 10+ knodes per move
-
- Posts: 151
- Joined: Thu Nov 12, 2009 6:31 pm
Re: Computer Chess Progress: Stockfish 7 vs Ruffian 1.0.5
The strength of stockfish is not only the result of algorithms, but mostly the result of fine tuning and massive testing at hyperbullet time control, which was not possible to achieve with 20 year old hardware.fierz wrote: So I think I'll claim that this backs up my original claim of 750 elo over 20 years due to software alone, and that this would theoretically have been possible 20 years ago too - the algorithms just were not there. It would be nice to try even older top engines (Shredder or Fritz 5.32) but I have no access to them.
Richard
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Computer Chess Progress: Stockfish 7 vs Ruffian 1.0.5
Did they try hyper-bullet back then? I am not aware of any tool facilitating that. Even depth=1 testing can be of some use, and I am again not aware of ancient tries, it could have been done even in 1960. In fact my guess is that they were very sloppy testers, and this is their fault, CC is an empirical field.abulmo wrote: massive testing at hyperbullet time control, which was not possible to achieve with 20 year old hardware.
-
- Posts: 1564
- Joined: Thu Jul 16, 2009 10:47 am
- Location: Almere, The Netherlands
Re: Computer Chess Progress: Stockfish 7 vs Ruffian 1.0.5
I keep a lot of engines from that period in a backup, most of the stronger ones are CB-dll format and not UCI compliant, I guess the only way to use them is with a CB-GUI.fierz wrote: It would be nice to try even older top engines (Shredder or Fritz 5.32) but I have no access to them.
-
- Posts: 710
- Joined: Sat Dec 06, 2014 1:53 pm
Re: Computer Chess Progress: Stockfish 7 vs Ruffian 1.0.5
2002 Nov. Version 1.05 o It's basically the same as v1.0.1 with some bugfixesfierz wrote:So I ended up running Ruffian 1.05 (2002 if I am not mistaken, not the very best engine around then, but in the top group)
2002-09-29 Version 1.0.1 o Fixed broken UCI interface. Ponder on should work now.
2002-09-22 Version 1.0.0 o First public release.
Offical results
---------------
2003, 5th CCT, Internet Champion 7/9 points, ranking 1 of 45
2003, 23rd Dutch Open, Champion 8.5/11 points, ranking 1 of 14
Did you disable supplied "Ruffian.bok" D=26 book?
-
- Posts: 151
- Joined: Thu Nov 12, 2009 6:31 pm
Re: Computer Chess Progress: Stockfish 7 vs Ruffian 1.0.5
Yes, sloppiness may be in cause. I just randomly open a typical paper of this era:Laskos wrote:Did they try hyper-bullet back then? I am not aware of any tool facilitating that. Even depth=1 testing can be of some use, and I am again not aware of ancient tries, it could have been done even in 1960. In fact my guess is that they were very sloppy testers, and this is their fault, CC is an empirical field.abulmo wrote: massive testing at hyperbullet time control, which was not possible to achieve with 20 year old hardware.
"Each match consisted of 40 games, with the time controls
set to 40 moves in 5 minutes.[...] The preliminary results indicate that
this method has some potential and is definitely worth
refining and experimenting with, but more study is def-
initely needed to reach a final verdict." (1)
The "preliminary result" was 24 - 16, which of course means nothing.
However, the heart of the problem was the lack of inexpensive fast multicore hardware 20 years ago. To believe that a 15knodes capable engine would have been able to play 1500 nodes in 0.1 sec (average time per move in hyperbullet games) is a misconception. in 0.1s, a program might have just initialize its hashtable and its eval, with no time remained to do the search.
--
(1) Bjornsson, Y. and Marsland, T. A. (2000b). Selective
depth-first search methods. In van den Herik, H. J. and Iida, H., editors,
Games in AI Research, pages 31–46.
Richard
-
- Posts: 179
- Joined: Fri Feb 14, 2014 10:53 pm
- Location: the Netherlands
Re: Computer Chess Progress: Stockfish 7 vs Ruffian 1.0.5
For a long time there was rather the common belief that plenty of clever tricks were left to be discovered. And that clever secret programming and search tricks along with adding tons of specific knowledge would lead to the better programs.
You do not resort to endless testing if you do not have to. It sucks the creative fun right out of computerchess.
Still around 2005 when I joined some tournaments in Leiden all the talk was crazy creative ideas, not endless tuning.
After Fruit it became more apparant that maybe the strong guys did not have all these clever crazy tricks and brilliant evaluationfunctions. But I'm sure a lot of guys were already aware of that long before then and that the Shredders, Ruffians etc. were a result of that.
You do not resort to endless testing if you do not have to. It sucks the creative fun right out of computerchess.
Still around 2005 when I joined some tournaments in Leiden all the talk was crazy creative ideas, not endless tuning.
After Fruit it became more apparant that maybe the strong guys did not have all these clever crazy tricks and brilliant evaluationfunctions. But I'm sure a lot of guys were already aware of that long before then and that the Shredders, Ruffians etc. were a result of that.
-
- Posts: 4367
- Joined: Fri Mar 10, 2006 5:23 am
- Location: http://www.arasanchess.org
Re: Computer Chess Progress: Stockfish 7 vs Ruffian 1.0.5
Yeah, I agree: I think it was trial and error back then and some sloppy testing. I used to run an overnight set of test suites. But that was a quite imperfect measure of strength. Still, with the hardware I had then, I could not run a useful series of matches in the same time period. Single core processors were the norm then.
--Jon
--Jon