SSDF Rating List 2013-03-23

[IanO]

(It is rather quaint that they continue to include dead projects Naum and Glaurung, though!)

Not to mention Shredder 8. About time to bump that one I think.

--Jon

Shredder 12 is in there, 7th place.

Personally, I'm glad they are still around to rate the new dedicateds (Phoenix) and handhelds (pocket fritz, hiarcs, genius, etc.)

[geots]

Where is Houdini , Stockfish , critter ?

I think they only include original programs in their testing.

That would confirm that Rybka is original then, especially as SSDF is the only rating list recognised by the Hiarcs forum.

You should know better than to use/abuse rating lists for declaring engines original or not .
Like other people in this thread I don't understand the motivation of the SSDF people to produce a rating list that has no relevance whatsoever for today's computer chess scene.

Robert

I am not sure what their motivations are or are not. Two things I would point out: I don't care if they have been the standard since dinosaurs walked the earth- Hiarcs that close to Rybka 4 in a rating list is such pure and total bullshit I wouldn't lower myself to discuss it with anyone. Secondly, they don't look too impressed with Levy and the "Chewing Gum Association".



gts

[Nelson Hernandez]

I wouldn't interpret SSDF's rating list as intending mischief. Putting things into proper historical context: it was the first rating list to be developed, and for a time (I'm talking 2004 and earlier) it was the only one around that had a transparent methodology and a palpable sense of impartiality. What happened circa 2005 was that CCRL and CEGT supplanted SSDF, which then steadily fell into ever-greater obsolescence. At this point SSDF is so out of synch with the current state of affairs that they cannot be taken seriously as a promulgator of ratings and rankings. Most annoying to me, the site where one might download the small number of test games that they do play hasn't been updated since July 2012.

[gerold]

Thanks for all the hard work you guys do with the original engines. Much appreciated. Almost the only pure engine list available.

Happy testing,
Gerold.

[S.Taylor]

Thanks for all the hard work you guys do with the original engines. Much appreciated. Almost the only pure engine list available.

Happy testing,
Gerold.

OK, so you are saying there is a proven difference? (and that the "clean" engines remain original and have nothing to do with the clones).

[geots]

Code: Select all

     THE SSDF RATING LIST 2013-03-23   %123376 games played by  312 computers							
                                           Rating   +     -  Games   Won  Oppo
                                           ------  ---   --- -----   ---  ----
   1 Deep Rybka 4 x64  2GB Q6600 2,4 GHz     3212   27   -25   930   75%  3022
   2 Deep Hiarcs 14 2GB Q6600 2,4 GHz        3205   29   -27   716   71%  3049
   3 Deep Rybka 3 x64  2GB Q6600 2,4 GHz     3202   24   -22  1251   77%  2991
   4 Naum 4.2 MP x64 2GB Q6600 2,4 GHz       3152   24   -23   919   64%  3050
   5 Naum 4 x64  2GB Q6600 2,4 GHz           3137   25   -23  1030   74%  2960
   6 Deep Junior 13.3 2GB x64 Q6600 2,4 GHz  3123   25   -25   760   57%  3074
   7 Deep Shredder 12 x64 2GB Q6600 2,4 GHz  3111   20   -19  1381   65%  3003
   8 Spike 1.4 MP 2GB Q6600  2,4 GHz         3106   20   -19  1320   64%  3009
   9 Deep Hiarcs 13.2  2GB Q6600 2,4 GHz     3106   28   -27   632   60%  3034
  10 Hiarcs 13.1  2GB Q6600 2,4 GHz          3104   25   -24   788   60%  3037
  11 Deep Fritz 13 2GB Q6600 2,4 GHz         3095   28   -27   624   58%  3037
  12 Deep Fritz 12 2GB Q6600 2,4 GHz         3094   21   -21  1040   56%  3050
  13 Deep Rybka 3  256MB Athlon 1200 MHz     3078   39   -37   332   58%  3022
  14 Deep Junior 12 x64 2GB  Q6600 2,4 GHz   3074   24   -23   938   63%  2979
  15 Deep Fritz 11  2GB Q6600 2,4 GHz        3066   19   -18  1424   63%  2977
  16 Zappa Mexico II x64  2GB Q6600 2,4 GHz  3057   25   -25   776   59%  2992
  17 Naum 3.1 x64  2GB Q6600 2,4 GHz         3043   27   -26   692   58%  2989
  18 Deep Hiarcs 12  2GB Q6600 2,4 GHz       3021   18   -18  1391   54%  2990
  19 Deep Shredder 11 x64 2GB Q6600  2,4 GHz 3017   22   -22   968   52%  3004
  20 Glaurung 2.2 x64 MP 2GB Q6600 2,4 GHz   2994   21   -21  1085   57%  2942
  21 Hiarcs 11.2 MP  2GB  Q6600  2,4 GHz     2989   22   -22   963   49%  2993
  22 Naum 4  256MB Athlon 1200 MHz           2988   28   -28   614   50%  2985
  23 Shredder 12 256MB A1200 MHz             2977   32   -32   446   50%  2977
  24 Deep Junior 10.1  2GB  Q6600  2,4 GHz   2970   23   -23   886   46%  2998
  25 Fritz 12  256MB  A1200 MHz              2951   29   -28   620   64%  2853
  26 Rybka 2.3.1 Arena 256MB Athlon 1200 MHz 2918   22   -21  1004   52%  2906
  27 Jonny 4.0 MP 2GB Q6600 2,4 GHz          2904   29   -31   600   30%  3049
  28 Deep Fritz 8 2GB  Q6600  2,4 GHz        2904   23   -24   929   35%  3011
  29 Shredder 8 MP 2GB  Q6600  2,4 GHz       2887   24   -25   908   32%  3014
  30 Deep Junior 8  2GB Q6600 2,4 GHz        2862   24   -25   952   30%  3012
  31 CM King 3.5 x64 MP 2GB  Q6600 2,4 GHz   2858   26   -28   752   29%  3011
  32 Zap!Chess Zanzibar 256MB Athlon 1200 MH 2834   21   -21  1100   50%  2837
  33 Fruit 2.2.1  256MB Athlon 1200 MHz      2829   18   -18  1465   59%  2767
  34 Pocket Fritz 4 Naum 4.2 Ipaq 214        2803   70   -64   115   62%  2717
  35 Chess Tiger 2007  256MB Athlon 1200 MHz 2795   21   -22  1154   35%  2905
  36 Pocket Fritz 4 Hiarcs 13 Ipaq 214       2753   64   -58   140   65%  2648
  37 Pocket Fritz 3 Hiarcs Ipaq 214 624 MHz  2727   53   -50   182   57%  2679
  38 Revelation Shredder 12  XScale 500 MHz  2704   60   -58   140   56%  2664
  39 Pocket Shredder Ipaq 114 624 MHz        2700   55   -51   180   63%  2611
  40 CEBoard Fruit 2.3.1 XScale 400  400 MHz 2662   50   -49   195   56%  2623
  41 Hiarcs Palm Chess 12.1 T5               2650   63   -64   120   48%  2662
  42 Revelation Rybka 2.2  XScale 500 MHz    2629   47   -44   240   62%  2547
  43 Pocket Fritz 2 XScale 400 MHz           2511   42   -42   265   52%  2495
  44 Pocket Fritz 3 Glaurung 2.1 Ipaq 614C   2502   59   -67   140   33%  2629
  45 Resurrection Rybka 2.2 StrongARM 203 MH 2485   43   -42   260   51%  2478
  46 Resurrection Fruit '05 StrongARM 203 MH 2393   67   -62   120   60%  2320
  47 Hiarcs 9.5a/9.6 Palm TungstenE OMAP 126 2393   34   -35   400   45%  2427
  48 CEBoard Crafty 2004 HP RX4240  400 MHz  2373   48   -50   200   43%  2424
  49 R30 v. 2.5                              2269   42   -39   323   67%  2143
  50 Palm Tiger 2009 Tung C  400 MHz         2238   54   -56   160   44%  2280



 2 Deep Hiarcs 14 2GB Q6600 2,4 GHz, 3205											
DRybka4 Q6600  24,5-15,5  DRybka3 Q6600   19,5-22,5  Naum4.2 Q6600   25,5-18,5
DJun133 Q6600	  21-21    DShre12 Q6600   25,5-18,5  Spike14 Q6600     30-12
Hia13.2 Q6600	24,5-15,5  DFrit13 Q6600     28-12    DFrit12 Q6600   61,5-24,5
DJuni12 Q6600	25,5-14,5  DFrit11 Q6600     31-9     Glaurung Q660   35,5-4,5
Jonny 4 Q6600	33,5-6,5   Hiar111 A1200   41,5-6,5   DJunio8 Q6600   36,5-3,5
CT 2007 A1200	  45-3								

 6 Deep Junior 13.3 2GB x64 Q6600 2,4 GHz, 3123											
DRybka4 Q6600   16,5-23,5  DHiar14 Q6600     21-21    DRybka3 Q6600   17,5-22,5
Naum4.2 Q6600   36,5-55,5  DShre12 Q6600   23,5-18,5  Spike14 Q6600   20,5-19,5
Hia13.2 Q6600   20,5-19,5  Hiarc13 Q6600     22-20    DFrit13 Q6600   22,5-17,5
DFrit12 Q6600   33,5-26,5  DJuni12 Q6600     21-19    DFrit11 Q6600     19-21
DShre11 Q6600   28,5-11,5  Jonny 4 Q6600   34,5-5,5   King3.5 Q6600     32-8
CT 2007 A1200   38,5-3,5

 11 Deep Fritz 13 2GB Q6600 2,4 GHz, 3095											
DRybka4 Q6600   13,5-26,5  DHiar14 Q6600     12-28     Naum4.2 Q6600  18,5-25,5
DJun133 Q6600   17,5-22,5  DShre12 Q6600   20,5-19,5   Spike14 Q6600    17-23
Hia13.2 Q6600     20-20    Hiarc13 Q6600   20,5-19,5   DFrit12 Q6600  17,5-22,5
DJuni12 Q6600   22,5-17,5  Jonny 4 Q6600     36-4      DFritz8 Q6600  28,5-16,5
DJunio8 Q6600     74-11    CT 2007 A1200   45,5-4,5


Here are the latest version of the SSDF Rating List. Since last time 
we have tested three new programs on our Q6600 2,4 GHz hardware. 

Mark Uniackes WCSC-winner of 2011 - Deep Hiarcs 14, is our new number 
two on the list with 3205, an improvement of 99 points from Deep 
Hiarcs 13.2!

Amir Ban and Shay Bushinsky's program, the WCCC-winner of 2011 - Deep 
Junior 13.3, takes the sixth place on our latest list. Their 
improvement is 49 points from the Deep Junior 12-version. We have 
tested the 64-bit version of the program.

The last of our tested program this time is Deep Fritz 13. It takes 
place eleven, 1 point in front of Deep Fritz 12.

For the next list we hope to have enough games with Revelation Hiarcs,
Chess Genius 3 iPhone 4 and to have some new entrants on both our 
Q6600 and A1200-level.

Everyone is entitled to mistakes, but I think this one should have been looked at hard before posting. I never question other testers' results on principle, but at any hash, any no. of cores, any time limit, and no matter the phase of the moon- it is cut and dried. Hiarcs 14 within 7 elo of Rybka 4 is without doubt the worst mistake I have ever seen in any rating list. Nothing is close. Case closed.


gts

[IWB]

Thanks for all the hard work you guys do with the original engines. Much appreciated. Almost the only pure engine list available.

Happy testing,
Gerold.

I know your intension was a different one but it is not "pure" at all. They test with books if available ('entity testing' - which is a reasonable 'pre-requirement'). H14 for sure has a very good book and looking at the opponent it played it is the newest book af all. Theoreticaly one has to look at all its games to see if it was the book that made the good performance (besides the long time control which for sure put everything closer).

This book testing explains a bit why they hesitate to test the other mentioned engines. They don't have a book to use and whatever they use it might be worse for that particular engine. (On the other hand I think they made exceptions already - Naum?!)

Bye
Ingo

[Terry McCracken]

Where is Houdini , Stockfish , critter ?

I think they only include original programs in their testing.

That would confirm that Rybka is original then, especially as SSDF is the only rating list recognised by the Hiarcs forum.

You should know better than to use/abuse rating lists for declaring engines original or not .
Like other people in this thread I don't understand the motivation of the SSDF people to produce a rating list that has no relevance whatsoever for today's computer chess scene.

Robert

I am not sure what their motivations are or are not. Two things I would point out: I don't care if they have been the standard since dinosaurs walked the earth- Hiarcs that close to Rybka 4 in a rating list is such pure and total bullshit I wouldn't lower myself to discuss it with anyone. Secondly, they don't look too impressed with Levy and the "Chewing Gum Association".



gts

Hiarcs is that close to Rybka, as Rybka is stagnating and Hiarcs has been progressing steadily.
Who cares what body impresses them. The ICGA is the top organising body in computer chess.

[Graham Banks]

Hiarcs is that close to Rybka, as Rybka is stagnating and Hiarcs has been progressing steadily.

Hiarcs is making good progress, but I think that most knowledgeable enthusiasts would concede that on naked engine strength alone, it's still nowhere near Rybka quite yet.
It's own opening book is superb though and makes a huge difference to its performance when used, probably more so than with any other engine.

[gerold]

Hiarcs is that close to Rybka, as Rybka is stagnating and Hiarcs has been progressing steadily.

Hiarcs is making good progress, but I think that most knowledgeable enthusiasts would concede that on naked engine strength alone, it's still nowhere near Rybka quite yet.
It's own opening book is superb though and makes a huge difference to its performance when used, probably more so than with any other engine.

Maybe run 1000 games without a book for Rybka and Hiarcs and see how they compare.

Terry is right Rybka has been dead for a long time. Where as Hiarcs continues to improve. Looks like Hiarcs will be the top good engine for awhile.

Best,
Gerold.

P.S. O.T. i Wonder when the clones will get smart and start their own world champ matches.