ChessUSA.com TalkChess.com
Hosted by Your Move Chess & Games
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Elo versus speed
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions Flat
View previous topic :: View next topic  
Author Message
Peter Österlund



Joined: 19 Apr 2010
Posts: 141
Location: Sweden

PostPost subject: Elo versus speed    Posted: Mon Apr 02, 2012 7:50 pm Reply to topic Reply with quote

Background

I recently converted my java chess engine CuckooChess to C++ (and renamed it to Texel). As a result the engine became approximately twice as fast. Initial tests suggest that the elo gain is much larger than what would have been expected according to the old rule that you gain 50-70 elo points for each doubling of the speed.

In an attempt to understand what is going on I played a time odds match using three different engines.

Match conditions

* Three engines tested: stockfish, crafty, texel
* Each engine plays at three different time controls:
1: 60 moves in 6 seconds
2: 60 moves in 12 seconds
3: 60 moves in 24 seconds
* Ponder off
* One CPU
* 128MB transposition table
* Openings selected randomly from a set of 7316 book lines, each line 10 ply deep.
* Each opening line played twice with colors reversed second time.
* Three test computers used, all running 64 bit linux.
- Core2 Quad 2.4GHz
- Core i7 870, 2.93GHz
- Core2 Duo, 2.6GHz
* Tournament where each engine plays all other engines.
* Total number of games: 14976
* Total number of games per engine: 3328
* Total number of games between each pair of engines: 416
* Total number of time losses: 1

Results

Rating computations were performed using bayeselo. All 14976 games together gives the following rating table:
Code:

Rank Name         Elo    +    - games score oppo. draws
   1 stockfish3   372   14   14  3328   91%   -47   14%
   2 stockfish2   228   12   11  3328   78%   -28   21%
   3 stockfish1    46   11   10  3328   56%    -6   23%
   4 texel3        32   11   11  3328   55%    -4   21%
   5 crafty3       30   11   10  3328   55%    -4   23%
   6 crafty2      -76   11   10  3328   41%     9   23%
   7 texel2      -116   10   11  3328   36%    14   21%
   8 crafty1     -222   11   12  3328   23%    28   19%
   9 texel1      -294   12   13  3328   16%    37   14%

From the table the elo increase when doubling the thinking time can be calculated:
Code:

            1->2  2->3
stockfish   182   144
crafty      146   106
texel       178   148

To see how the individual engines performed against each other, all games involving a given pair of engines were extracted and bayeselo was used to calculate the rating difference given those games. This was repeated for all pairs of engines. The result was:
Code:

                 s3   s2   s1   c3   c2   c1   t3   t2   t1
stockfish3        0  134  344  330  448  576  334  500  624
stockfish2     -134    0  178  224  336  396  168  342  470
stockfish1     -344 -178    0   16  108  268   50  160  308
crafty3        -330 -224  -16    0  114  270   -2  136  324
crafty2        -448 -336 -108 -114    0  174  -98   42  190
crafty1        -576 -396 -268 -270 -174    0 -228 -104   76
texel3         -334 -168  -50    2   98  228    0  162  372
texel2         -500 -342 -160 -136  -42  104 -162    0  196
texel1         -624 -470 -308 -324 -190  -76 -372 -196    0

Taking differences in the above table gives:
Code:

                 s3   s2   s1   c3   c2   c1   t3   t2   t1
stockfish 2->3  134  134  166  106  112  180  166  158  154
stockfish 1->2  210  178  178  208  228  128  118  182  162
crafty    2->3  118  112   92  114  114   96   96   94  134
crafty    1->2  128   60  160  156  174  174  130  146  114
texel     2->3  166  174  110  138  140  124  162  162  176
texel     1->2  124  128  148  188  148  180  210  196  196

Discussion

I believe the "50-70 elo increase per speed doubling" estimate originally comes from "How Computers Play Chess" by David Levy and Monty Newborn in 1991. I don't have that book, so I don't know what data they based the estimate on. I guess they used longer time controls than I did, but given 20 years of hardware improvement, it is not unlikely that my tests generated larger search trees anyway.

In my last table, most values are significantly larger than 50-70. In fact only one value (crafty 1->2 vs stockfish2) falls within the 50-70 range. It is worth noting that each value in that table is calculated from only 416 games, so the error margin according to bayeselo is about 30. However, if the true values were within 50-70, getting the result I got would be extremely unlikely.

Are there any newer estimates of elo versus speed? I would not be surprised if todays engines with recursive null moves, late move reductions and other pruning techniques behave quite differently from the ones from 20 years ago.
_________________
Peter Osterlund - peterosterlund2@gmail.com
http://web.comhem.se/petero2home
Back to top
View user's profile Send private message Send e-mail Visit poster's website MSN Messenger
Display posts from previous:   
Subject Author Date/Time
Elo versus speed Peter Österlund Mon Apr 02, 2012 7:50 pm
      Re: Elo versus speed Larry Kaufman Mon Apr 02, 2012 8:07 pm
            Re: Elo versus speed Uri Blass Mon Apr 02, 2012 8:28 pm
                  Re: Elo versus speed Vincent Diepeveen Tue Apr 03, 2012 3:34 am
      Re: Elo versus speed Daniel Shawul Tue Apr 03, 2012 9:07 am
            Re: Elo versus speed Ed Schroder Tue Apr 03, 2012 1:22 pm
                  Re: Elo versus speed Daniel Shawul Tue Apr 03, 2012 2:23 pm
                  Re: Elo versus speed Vincent Diepeveen Tue Apr 03, 2012 4:04 pm
      Re: Elo versus speed Adam Hair Tue Apr 03, 2012 10:29 am
      Re: Elo versus speed Ed Schroder Tue Apr 03, 2012 1:14 pm
            Re: Elo versus speed Vincent Diepeveen Tue Apr 03, 2012 4:03 pm
      Re: Elo versus speed Jacob Børs Lind Fri Apr 06, 2012 7:05 pm
            Re: Elo versus speed Vincent Diepeveen Fri Apr 06, 2012 10:09 pm
                  Re: Elo versus speed Jacob Børs Lind Sat Apr 07, 2012 12:15 am
                        Re: Elo versus speed Vincent Diepeveen Sat Apr 07, 2012 9:03 am
                  Re: Elo versus speed Ed Schroder Sun Apr 08, 2012 11:16 am
                        Re: Elo versus speed Vincent Diepeveen Sun Apr 08, 2012 2:43 pm
                              Re: Elo versus speed Uri Blass Sun Apr 08, 2012 3:58 pm
                                    Re: Elo versus speed Vincent Diepeveen Sun Apr 08, 2012 7:58 pm
                              Re: Elo versus speed Ed Schroder Sun Apr 08, 2012 11:30 pm
                                    Re: Elo versus speed Vincent Diepeveen Mon Apr 09, 2012 12:47 am
                                          Re: Elo versus speed Ricardo Barreira Mon Apr 09, 2012 1:56 am
                                                Re: Elo versus speed Vincent Diepeveen Mon Apr 09, 2012 8:39 am
                                                      Re: Elo versus speed Uri Blass Mon Apr 09, 2012 4:56 pm
                                                            Re: Elo versus speed Vincent Diepeveen Tue Apr 10, 2012 8:25 am
                                                                  Re: Elo versus speed Uri Blass Tue Apr 10, 2012 8:58 pm
                                                                        Re: Elo versus speed Vincent Diepeveen Wed Apr 11, 2012 10:37 am
                                                                              Re: Elo versus speed Uri Blass Wed Apr 11, 2012 4:18 pm
      Re: Elo versus speed Peter Österlund Sun Apr 08, 2012 9:37 am
            Re: Elo versus speed Ian Osgood Mon Apr 09, 2012 6:19 pm
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum




Powered by phpBB © 2001, 2005 phpBB Group
Enhanced with Moby Threads