ChessUSA.com TalkChess.com
Hosted by Your Move Chess & Games
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Elo Increase per Doubling
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions Flat
View previous topic :: View next topic  
Author Message
Don Dailey



Joined: 29 Apr 2008
Posts: 4323
PostPost subject: Re: Elo Increase per Doubling    Posted: Tue May 08, 2012 5:32 pm Reply to topic Reply with quote
bob wrote:
Don wrote:
Don wrote:
Rebel wrote:
Adam Hair wrote:
I do think I will revisit both of these types of measurements. Also, Miguel has suggested to me to measure the increase in Elo when the number of nodes is doubled.

Good idea, just one but: time control is killed. But since that's true for both engines.....

Is there no interface that allows each engine its own time control ?


I'm running a test of pure nodes doubling now. I need my computer for other things so this will be just a few hundred games. Here is what I have so far (I'm still running) :

Code:
Rank Name    Elo      +      -    games   score   oppo.   draws
   1 10    4607.7   96.1   96.1     169   94.1%  3851.3   11.8%
   2 09    4498.6   84.1   84.1     169   87.9%  3861.6   17.2%
   3 08    4400.2   81.3   81.3     170   80.0%  3875.8   12.9%
   4 07    4270.4   79.3   79.3     170   70.9%  3888.8   15.9%
   5 06    4135.0   81.4   81.4     170   61.8%  3902.4   14.1%
   6 05    3969.2   84.4   84.4     170   51.2%  3918.9    5.9%
   7 04    3757.1   84.5   84.5     170   37.4%  3940.1   11.2%
   8 03    3654.1   87.0   87.0     170   30.3%  3950.5   11.2%
   9 02    3543.2   89.7   89.7     170   23.5%  3961.5   10.6%
  10 01    3323.0  108.9  108.9     170   12.1%  3983.6    5.3%
  11 00    3000.0  179.5  179.5     170    1.5%  4015.9    0.6%


      TIME       RATIO    log(r)     NODES    log(r)  ave DEPTH    GAMES   PLAYER
 ---------  ----------  --------  --------  --------  ---------  -------   --
    0.0022       1.000     0.000     0.001     0.000     3.1752      170   00
    0.0028       1.302     0.264     0.001     0.685     4.2193      170   01
    0.0046       2.122     0.752     0.002     1.378     5.3787      170   02
    0.0082       3.748     1.321     0.004     2.071     6.5317      170   03
    0.0158       7.263     1.983     0.008     2.765     7.6667      170   04
    0.0326      14.939     2.704     0.016     3.457     8.7110      170   05
    0.0577      26.458     3.276     0.033     4.151     9.7802      170   06
    0.1052      48.235     3.876     0.066     4.844    10.8375      170   07
    0.1926      88.292     4.481     0.131     5.537    11.8340      170   08
    0.3901     178.827     5.186     0.262     6.230    13.0136      169   09
    0.7313     335.232     5.815     0.524     6.923    14.0442      169   10


As Ed observes you do lose time control but it's the same with fixed depth. The average depth is interesting too, we get a little more than a ply for each doubling of nodes.

For reference, version 00 is 512 nodes and each subsequent level doubles that.

I'll report again later with the delta's and a graph.

Don



P.S. I have bayeselo set to use confidence of 98% so my error margins will be larger than what is normal.


My personal feeling is that this tends to exaggerate the rating gain. You are basically playing A vs A' where the ONLY difference is 2x speed. Identical evals, identical searches, identical time controls. This won't hold true against a suite of opponents, is my opinion. It also tends to break some important things. For example, Crafty tries very hard to always finish the current ply-1 move it is searching, before aborting on time, just in case this is fixing to fail high (or low). With fixed nodes, it can't do that....

I hate nodes/depth as a search limit for trying to measure anything to do with rating (improvement)


I don't know any way to really get an accurate rating, how you test influences things. For example only testing against other computers is not accurate either. The node testing I have already proved hurts the ELO substantially as a good time control algorithm is worth quite a bit. However I don't think it has any effect on the validity of this test. I cannot imagine that if you sudden death games instead of fixed nodes I would find that program improve MORE with depth for example.

I have also found that playing programs in round robin fashion where the ELO differences are this huge has a very strong compressing effect on the ratings. I have done similar experiments where I only let a player play up or down 2 or 3 levels and the rating spread is significantly larger.

I think a lot of this has to do with the contempt. I have seen games Komodo should NEVER have to suffer a draw because the opponent is over 1000 ELO weaker but Komodo is playing black and right out of book finds a way to force a draw!

In the test I just ran there was 1 draw of 10 vs 00 even though the difference is a whopping 1627 ELO! The game was very short - draw by repetition on whites 16th move.
_________________
"Your superior intellect is no match for our puny weapons." -Kang and Kodos
Back to top
View user's profile Send private message Send e-mail
Display posts from previous:   
Subject Author Date/Time
Elo Increase per Doubling Adam Hair Mon May 07, 2012 11:40 am
      Re: Elo Increase per Doubling Julien MARCEL Mon May 07, 2012 12:04 pm
            Re: Elo Increase per Doubling Adam Hair Mon May 07, 2012 4:48 pm
                  Re: Elo Increase per Doubling Julien MARCEL Mon May 07, 2012 4:54 pm
            Re: Elo Increase per Doubling Adam Hair Mon May 07, 2012 8:50 pm
                  Re: Elo Increase per Doubling Julien MARCEL Mon May 07, 2012 9:00 pm
                        Re: Elo Increase per Doubling Mark Pearce Thu May 10, 2012 4:43 pm
      Re: Elo Increase per Doubling H.G.Muller Mon May 07, 2012 12:13 pm
            Re: Elo Increase per Doubling Michel Van den Bergh Mon May 07, 2012 2:01 pm
            Re: Elo Increase per Doubling Adam Hair Mon May 07, 2012 5:12 pm
                  Re: Elo Increase per Doubling Don Dailey Mon May 07, 2012 6:40 pm
                        Re: Elo Increase per Doubling Robert Wagner Campos Tue May 08, 2012 12:38 am
                              Re: Elo Increase per Doubling Don Dailey Tue May 08, 2012 3:32 am
                                    Re: Elo Increase per Doubling Adam Hair Tue May 08, 2012 10:02 am
                                          Re: Elo Increase per Doubling Julien MARCEL Tue May 08, 2012 10:55 am
                                                Re: Elo Increase per Doubling Adam Hair Tue May 08, 2012 8:08 pm
                                          Re: Elo Increase per Doubling Don Dailey Tue May 08, 2012 11:29 am
                                          Re: Elo Increase per Doubling Kai Laskos Tue May 08, 2012 11:59 am
                                                Re: Elo Increase per Doubling Adam Hair Tue May 08, 2012 8:13 pm
                                                      Re: Elo Increase per Doubling Kai Laskos Tue May 08, 2012 10:21 pm
                                                            Re: Elo Increase per Doubling Don Dailey Tue May 08, 2012 10:52 pm
                                                            Re: Elo Increase per Doubling Adam Hair Tue May 08, 2012 11:14 pm
                                                                  Re: Elo Increase per Doubling Kai Laskos Wed May 09, 2012 12:20 am
                                                                        Re: Elo Increase per Doubling Adam Hair Wed May 09, 2012 2:44 am
                                                                              Re: Elo Increase per Doubling H.G.Muller Wed May 09, 2012 8:05 am
                                                                                    Re: Elo Increase per Doubling Adam Hair Wed May 09, 2012 5:12 pm
                                                                                          Re: Elo Increase per Doubling Larry Kaufman Thu May 10, 2012 3:11 am
                                                                                          Re: Elo Increase per Doubling Don Dailey Thu May 10, 2012 5:03 am
                                                                                          Re: Elo Increase per Doubling Larry Kaufman Thu May 10, 2012 5:19 am
                                                                                          Re: Elo Increase per Doubling Don Dailey Thu May 10, 2012 12:43 pm
                                                                                          Re: Elo Increase per Doubling Peter Österlund Thu May 10, 2012 4:59 pm
                                                                                          Re: Elo Increase per Doubling Julien MARCEL Thu May 10, 2012 5:09 pm
                                                                                          Re: Elo Increase per Doubling Don Dailey Thu May 10, 2012 5:41 pm
                                                                                          Re: Elo Increase per Doubling Don Dailey Thu May 10, 2012 5:46 pm
                                          Re: Elo Increase per Doubling Ed Schroder Tue May 08, 2012 12:09 pm
                                                Re: Elo Increase per Doubling Don Dailey Tue May 08, 2012 12:10 pm
                                                      Re: Elo Increase per Doubling Ed Schroder Tue May 08, 2012 12:28 pm
                                                            Re: Elo Increase per Doubling Don Dailey Tue May 08, 2012 12:36 pm
                                                                  Re: Elo Increase per Doubling Ed Schroder Tue May 08, 2012 1:15 pm
                                                                        Re: Elo Increase per Doubling Don Dailey Tue May 08, 2012 2:37 pm
                                                                              Re: Elo Increase per Doubling Julien MARCEL Tue May 08, 2012 3:15 pm
                                                                                    Re: Elo Increase per Doubling Don Dailey Tue May 08, 2012 6:15 pm
                                                                                          Re: Elo Increase per Doubling Peter Österlund Tue May 08, 2012 8:06 pm
                                                                                          Re: Elo Increase per Doubling Don Dailey Tue May 08, 2012 8:25 pm
                                                                                          Re: Elo Increase per Doubling Peter Österlund Tue May 08, 2012 9:00 pm
                                                                                          Re: Elo Increase per Doubling Julien MARCEL Tue May 08, 2012 9:02 pm
                                                                                          Re: Elo Increase per Doubling Don Dailey Tue May 08, 2012 9:06 pm
                                                                                          Re: Elo Increase per Doubling Don Dailey Wed May 09, 2012 4:26 pm
                                                                                          Re: Elo Increase per Doubling Peter Österlund Wed May 09, 2012 4:52 pm
                                                                                          Re: Elo Increase per Doubling Don Dailey Wed May 09, 2012 5:19 pm
                                                                                          Re: Elo Increase per Doubling Don Dailey Thu May 10, 2012 1:41 pm
                                                                                          Re: Elo Increase per Doubling Don Dailey Thu May 10, 2012 4:52 pm
                                                                                          Re: Elo Increase per Doubling Adam Hair Thu May 10, 2012 9:07 pm
                                                                                          Re: Elo Increase per Doubling Don Dailey Thu May 10, 2012 10:55 pm
                                                                                          Re: Elo Increase per Doubling Adam Hair Fri May 11, 2012 12:46 am
                                                                                          Re: Elo Increase per Doubling Adam Hair Fri May 11, 2012 1:06 am
                                                                                          Re: Elo Increase per Doubling Don Dailey Wed May 09, 2012 5:00 pm
                                                                                          Re: Elo Increase per Doubling Adam Hair Tue May 08, 2012 9:05 pm
                                                                                          Re: Elo Increase per Doubling Don Dailey Tue May 08, 2012 9:18 pm
                                                                                          Re: Elo Increase per Doubling Adam Hair Tue May 08, 2012 11:26 pm
                                                                                          Re: Elo Increase per Doubling Don Dailey Wed May 09, 2012 2:50 am
                                                                                          Re: Elo Increase per Doubling Adam Hair Tue May 15, 2012 3:17 am
                                                                                    Re: Elo Increase per Doubling Don Dailey Tue May 08, 2012 6:23 pm
                                          Re: Elo Increase per Doubling Ed Schroder Tue May 08, 2012 12:18 pm
                                                Re: Elo Increase per Doubling Don Dailey Tue May 08, 2012 12:25 pm
                                                      Re: Elo Increase per Doubling Don Dailey Tue May 08, 2012 12:27 pm
                                                            Re: Elo Increase per Doubling Robert Hyatt Tue May 08, 2012 3:36 pm
                                                                  Re: Elo Increase per Doubling Don Dailey Tue May 08, 2012 5:32 pm
      Re: Elo Increase per Doubling Daniel Shawul Fri May 11, 2012 5:48 pm
            Re: Elo Increase per Doubling Don Dailey Fri May 11, 2012 6:30 pm
                  Re: Elo Increase per Doubling Julien MARCEL Fri May 11, 2012 6:52 pm
                        Re: Elo Increase per Doubling Don Dailey Fri May 11, 2012 7:05 pm
                  Re: Elo Increase per Doubling Daniel Shawul Fri May 11, 2012 7:08 pm
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum




Powered by phpBB © 2001, 2005 phpBB Group
Enhanced with Moby Threads