| View previous topic :: View next topic |
| Author |
Message |
Peter Österlund
Joined: 19 Apr 2010 Posts: 141 Location: Sweden
|
Post subject: Elo versus speed Posted: Mon Apr 02, 2012 7:50 pm |
|
|
Background
I recently converted my java chess engine CuckooChess to C++ (and renamed it to Texel). As a result the engine became approximately twice as fast. Initial tests suggest that the elo gain is much larger than what would have been expected according to the old rule that you gain 50-70 elo points for each doubling of the speed.
In an attempt to understand what is going on I played a time odds match using three different engines.
Match conditions
* Three engines tested: stockfish, crafty, texel
* Each engine plays at three different time controls:
1: 60 moves in 6 seconds
2: 60 moves in 12 seconds
3: 60 moves in 24 seconds
* Ponder off
* One CPU
* 128MB transposition table
* Openings selected randomly from a set of 7316 book lines, each line 10 ply deep.
* Each opening line played twice with colors reversed second time.
* Three test computers used, all running 64 bit linux.
- Core2 Quad 2.4GHz
- Core i7 870, 2.93GHz
- Core2 Duo, 2.6GHz
* Tournament where each engine plays all other engines.
* Total number of games: 14976
* Total number of games per engine: 3328
* Total number of games between each pair of engines: 416
* Total number of time losses: 1
Results
Rating computations were performed using bayeselo. All 14976 games together gives the following rating table:
| Code: |
Rank Name Elo + - games score oppo. draws
1 stockfish3 372 14 14 3328 91% -47 14%
2 stockfish2 228 12 11 3328 78% -28 21%
3 stockfish1 46 11 10 3328 56% -6 23%
4 texel3 32 11 11 3328 55% -4 21%
5 crafty3 30 11 10 3328 55% -4 23%
6 crafty2 -76 11 10 3328 41% 9 23%
7 texel2 -116 10 11 3328 36% 14 21%
8 crafty1 -222 11 12 3328 23% 28 19%
9 texel1 -294 12 13 3328 16% 37 14%
|
From the table the elo increase when doubling the thinking time can be calculated:
| Code: |
1->2 2->3
stockfish 182 144
crafty 146 106
texel 178 148
|
To see how the individual engines performed against each other, all games involving a given pair of engines were extracted and bayeselo was used to calculate the rating difference given those games. This was repeated for all pairs of engines. The result was:
| Code: |
s3 s2 s1 c3 c2 c1 t3 t2 t1
stockfish3 0 134 344 330 448 576 334 500 624
stockfish2 -134 0 178 224 336 396 168 342 470
stockfish1 -344 -178 0 16 108 268 50 160 308
crafty3 -330 -224 -16 0 114 270 -2 136 324
crafty2 -448 -336 -108 -114 0 174 -98 42 190
crafty1 -576 -396 -268 -270 -174 0 -228 -104 76
texel3 -334 -168 -50 2 98 228 0 162 372
texel2 -500 -342 -160 -136 -42 104 -162 0 196
texel1 -624 -470 -308 -324 -190 -76 -372 -196 0
|
Taking differences in the above table gives:
| Code: |
s3 s2 s1 c3 c2 c1 t3 t2 t1
stockfish 2->3 134 134 166 106 112 180 166 158 154
stockfish 1->2 210 178 178 208 228 128 118 182 162
crafty 2->3 118 112 92 114 114 96 96 94 134
crafty 1->2 128 60 160 156 174 174 130 146 114
texel 2->3 166 174 110 138 140 124 162 162 176
texel 1->2 124 128 148 188 148 180 210 196 196
|
Discussion
I believe the "50-70 elo increase per speed doubling" estimate originally comes from "How Computers Play Chess" by David Levy and Monty Newborn in 1991. I don't have that book, so I don't know what data they based the estimate on. I guess they used longer time controls than I did, but given 20 years of hardware improvement, it is not unlikely that my tests generated larger search trees anyway.
In my last table, most values are significantly larger than 50-70. In fact only one value (crafty 1->2 vs stockfish2) falls within the 50-70 range. It is worth noting that each value in that table is calculated from only 416 games, so the error margin according to bayeselo is about 30. However, if the true values were within 50-70, getting the result I got would be extremely unlikely.
Are there any newer estimates of elo versus speed? I would not be surprised if todays engines with recursive null moves, late move reductions and other pruning techniques behave quite differently from the ones from 20 years ago. _________________ Peter Osterlund - peterosterlund2@gmail.com
http://web.comhem.se/petero2home |
|
| Back to top |
|
 |
|
| Subject |
Author |
Date/Time |
Elo versus speed |
Peter Österlund |
Mon Apr 02, 2012 7:50 pm |
Re: Elo versus speed |
Larry Kaufman |
Mon Apr 02, 2012 8:07 pm |
Re: Elo versus speed |
Uri Blass |
Mon Apr 02, 2012 8:28 pm |
Re: Elo versus speed |
Vincent Diepeveen |
Tue Apr 03, 2012 3:34 am |
Re: Elo versus speed |
Daniel Shawul |
Tue Apr 03, 2012 9:07 am |
Re: Elo versus speed |
Ed Schroder |
Tue Apr 03, 2012 1:22 pm |
Re: Elo versus speed |
Daniel Shawul |
Tue Apr 03, 2012 2:23 pm |
Re: Elo versus speed |
Vincent Diepeveen |
Tue Apr 03, 2012 4:04 pm |
Re: Elo versus speed |
Adam Hair |
Tue Apr 03, 2012 10:29 am |
Re: Elo versus speed |
Ed Schroder |
Tue Apr 03, 2012 1:14 pm |
Re: Elo versus speed |
Vincent Diepeveen |
Tue Apr 03, 2012 4:03 pm |
Re: Elo versus speed |
Jacob Børs Lind |
Fri Apr 06, 2012 7:05 pm |
Re: Elo versus speed |
Vincent Diepeveen |
Fri Apr 06, 2012 10:09 pm |
Re: Elo versus speed |
Jacob Børs Lind |
Sat Apr 07, 2012 12:15 am |
Re: Elo versus speed |
Vincent Diepeveen |
Sat Apr 07, 2012 9:03 am |
Re: Elo versus speed |
Ed Schroder |
Sun Apr 08, 2012 11:16 am |
Re: Elo versus speed |
Vincent Diepeveen |
Sun Apr 08, 2012 2:43 pm |
Re: Elo versus speed |
Uri Blass |
Sun Apr 08, 2012 3:58 pm |
Re: Elo versus speed |
Vincent Diepeveen |
Sun Apr 08, 2012 7:58 pm |
Re: Elo versus speed |
Ed Schroder |
Sun Apr 08, 2012 11:30 pm |
Re: Elo versus speed |
Vincent Diepeveen |
Mon Apr 09, 2012 12:47 am |
Re: Elo versus speed |
Ricardo Barreira |
Mon Apr 09, 2012 1:56 am |
Re: Elo versus speed |
Vincent Diepeveen |
Mon Apr 09, 2012 8:39 am |
Re: Elo versus speed |
Uri Blass |
Mon Apr 09, 2012 4:56 pm |
Re: Elo versus speed |
Vincent Diepeveen |
Tue Apr 10, 2012 8:25 am |
Re: Elo versus speed |
Uri Blass |
Tue Apr 10, 2012 8:58 pm |
Re: Elo versus speed |
Vincent Diepeveen |
Wed Apr 11, 2012 10:37 am |
Re: Elo versus speed |
Uri Blass |
Wed Apr 11, 2012 4:18 pm |
Re: Elo versus speed |
Peter Österlund |
Sun Apr 08, 2012 9:37 am |
Re: Elo versus speed |
Ian Osgood |
Mon Apr 09, 2012 6:19 pm |
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|