ChessUSA.com TalkChess.com
Hosted by Your Move Chess & Games
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Observator bias or...
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions Flat
View previous topic :: View next topic  
Author Message
Robert Hyatt



Joined: 27 Feb 2006
Posts: 15819
Location: Birmingham, AL
PostPost subject: Re: Observator bias or...    Posted: Sat Jun 02, 2007 6:37 am Reply to topic Reply with quote
hgm wrote:
Alessandro Scotti wrote:
I remember since testing with Kiwi that results with 100 games are very unreliable. It sometimes happen that a version gets a bad start but gets better at the end of the long test. On the other hand, I had a version reach 64% after 100 games and finish with a disappointing 50% after 720 games... I will now increase the number to 800 and see if that brings some benefits (not much is expected though).

64% after 100 games between approximately equal engines is extreme: the standard error over 100 games should be 0.4/sqrt(100) = 4%, so a 14% deviation represents 3.5 sigma. This should happen on the average only 1 in ~4000 tries.

I noted a very strange effect when I was testing uMax in self play. The standard error over 100 games should be 4%, but when I played 1000 games between the same versions, and looked at the scores of the ten individual 100-game runs, these results deviated on the average much more from each other (and the final average result) than you would expect from the calculated standard error. This can only happen if the games are not independent! I can indeed not exclude this, as all the games were played in a single run, and were using the random seed the previous game ended with. So with a bad randomizer, if a single game repeats due to an equal or very close seed at the start of the game, it might imply that the following game repeats as well, destroying the independence of the game.

Whatever the cause, the effect was that the error in the win percentage was always a lot larger than you would expect based on the number of games.


My current testing methodology is to play 40 positions, once black, once white, and do this 32 times (64 games per position) and to repeat this against multiple opponents. This is giving pretty stable results and allows me to compare two versions with reasonable reliability. Anything less is not enough, based on a few hundred thousand games testing this. Smile
Back to top
View user's profile Send private message
Display posts from previous:   
Subject Author Date/Time
Observator bias or... Alessandro Scotti Tue May 29, 2007 6:25 pm
      Re: Observator bias or... Dann Corbit Tue May 29, 2007 6:33 pm
      Re: Observator bias or... ed Tue May 29, 2007 6:59 pm
            Re: Observator bias or... H.G.Muller Wed May 30, 2007 9:43 am
                  Re: Observator bias or... ed Wed May 30, 2007 3:02 pm
                        Re: Observator bias or... cwb Wed May 30, 2007 4:48 pm
                        Re: Observator bias or... Peter Fendrich Wed May 30, 2007 6:36 pm
                        Re: Observator bias or... Robert Hyatt Sun Jun 10, 2007 3:30 pm
      Re: Observator bias or... Uri Blass Wed May 30, 2007 10:17 am
            Re: Observator bias or... Alessandro Scotti Wed May 30, 2007 12:35 pm
                  Re: Observator bias or... Robert Hyatt Sat Jun 02, 2007 6:41 am
                        Re: Observator bias or... H.G.Muller Sat Jun 02, 2007 10:18 am
                              Re: Observator bias or... Robert Hyatt Sun Jun 03, 2007 12:54 am
                                    Re: Observator bias or... Uri Blass Sun Jun 03, 2007 5:44 am
                                          Re: Observator bias or... Robert Hyatt Sun Jun 03, 2007 8:34 pm
                                                Re: Observator bias or... H.G.Muller Mon Jun 04, 2007 9:52 am
                                                      Re: Observator bias or... Robert Hyatt Mon Jun 04, 2007 6:58 pm
                                                            Re: Observator bias or... H.G.Muller Tue Jun 05, 2007 2:24 pm
                                                                  Re: Observator bias or... Robert Hyatt Wed Jun 06, 2007 1:31 am
                                                                        Re: Observator bias or... Uri Blass Wed Jun 06, 2007 6:42 am
                                                                              Re: Observator bias or... Robert Hyatt Thu Jun 07, 2007 2:18 am
                                                                        Re: Observator bias or... H.G.Muller Thu Jun 07, 2007 2:20 pm
                                                                              Re: Observator bias or... Robert Hyatt Fri Jun 08, 2007 3:31 am
                                                                                    time for some real data Robert Hyatt Fri Jun 08, 2007 4:02 pm
                                                                                          Re: time for some real data H.G.Muller Fri Jun 08, 2007 4:51 pm
                                                                                          Re: time for some real data Robert Hyatt Sat Jun 09, 2007 1:43 am
                                    Re: Observator bias or... H.G.Muller Fri Jun 08, 2007 9:40 am
                                          Re: Observator bias or... Robert Hyatt Sun Jun 10, 2007 2:24 am
                              Re: Observator bias or... Charles Roberson Wed Jun 06, 2007 2:44 am
                                    Re: Observator bias or... Uri Blass Wed Jun 06, 2007 6:46 am
      Re: Observator bias or... Ron Murawski Wed May 30, 2007 8:26 pm
            Re: Observator bias or... Alessandro Scotti Wed May 30, 2007 8:31 pm
                  Re: Observator bias or... ed Wed May 30, 2007 11:50 pm
                        Re: Observator bias or... Dann Corbit Thu May 31, 2007 12:19 am
                              Re: Observator bias or... Dann Corbit Thu May 31, 2007 12:33 am
                                    Re: Observator bias or... Dann Corbit Thu May 31, 2007 12:40 am
                                          Re: Observator bias or... ed Thu May 31, 2007 9:40 am
                  Re: Observator bias or... H.G.Muller Thu May 31, 2007 11:02 am
                        Re: Observator bias or... Tony Thu May 31, 2007 12:04 pm
                              Re: Observator bias or... Uri Blass Thu May 31, 2007 12:51 pm
                                    Re: Observator bias or... Tony Thu May 31, 2007 12:55 pm
                        Re: Observator bias or... Alessandro Scotti Thu May 31, 2007 12:56 pm
                        Re: Observator bias or... Robert Hyatt Sat Jun 02, 2007 6:37 am
                        Re: Observator bias or... Eelco de Groot Sat Jun 02, 2007 11:15 pm
      Re: Observator bias or... Michael Sherwin Sun Jun 03, 2007 6:29 am
            Re: Observator bias or... Uri Blass Sun Jun 03, 2007 8:11 am
                  Re: Observator bias or... Eelco de Groot Sun Jun 03, 2007 9:07 am
                        Re: Observator bias or... Uri Blass Sun Jun 03, 2007 9:39 am
                        Re: Observator bias or... H.G.Muller Sun Jun 03, 2007 9:47 am
            Re: Observator bias or... Alessandro Scotti Sun Jun 03, 2007 8:36 am
                  Re: Observator bias or... Ron Murawski Sun Jun 03, 2007 5:50 pm
      Re: Observator bias or... MartinBryant Sun Jun 03, 2007 9:07 am
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum




Powered by phpBB © 2001, 2005 phpBB Group
Enhanced with Moby Threads