TalkChess.com
Hosted by Your Move Chess & Games

Author Message
Uri Blass

Joined: 08 Mar 2006
Posts: 5957
Location: Tel-Aviv Israel

Post subject: Re: Observator bias or...    Posted: Thu May 31, 2007 12:51 pm

Tony wrote:
hgm wrote:
 Alessandro Scotti wrote: I remember since testing with Kiwi that results with 100 games are very unreliable. It sometimes happen that a version gets a bad start but gets better at the end of the long test. On the other hand, I had a version reach 64% after 100 games and finish with a disappointing 50% after 720 games... I will now increase the number to 800 and see if that brings some benefits (not much is expected though).

64% after 100 games between approximately equal engines is extreme: the standard error over 100 games should be 0.4/sqrt(100) = 4%, so a 14% deviation represents 3.5 sigma. This should happen on the average only 1 in ~4000 tries.

I noted a very strange effect when I was testing uMax in self play. The standard error over 100 games should be 4%, but when I played 1000 games between the same versions, and looked at the scores of the ten individual 100-game runs, these results deviated on the average much more from each other (and the final average result) than you would expect from the calculated standard error. This can only happen if the games are not independent! I can indeed not exclude this, as all the games were played in a single run, and were using the random seed the previous game ended with. So with a bad randomizer, if a single game repeats due to an equal or very close seed at the start of the game, it might imply that the following game repeats as well, destroying the independence of the game.

Whatever the cause, the effect was that the error in the win percentage was always a lot larger than you would expect based on the number of games.

I think the math only works if P(win)=P(loose)=P(draw)=1/3 (which I doubt is the case)

Ed's code even assumes P(win,white)==P(win,black) which I doubt as well.

Tony

With bigger probability for white the variance is even smaller so result of 64% after 100 games is even less expected.

Uri
 Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First
Subject Author Date/Time
Alessandro Scotti Tue May 29, 2007 6:25 pm
Dann Corbit Tue May 29, 2007 6:33 pm
ed Tue May 29, 2007 6:59 pm
H.G.Muller Wed May 30, 2007 9:43 am
ed Wed May 30, 2007 3:02 pm
cwb Wed May 30, 2007 4:48 pm
Peter Fendrich Wed May 30, 2007 6:36 pm
Robert Hyatt Sun Jun 10, 2007 3:30 pm
Uri Blass Wed May 30, 2007 10:17 am
Alessandro Scotti Wed May 30, 2007 12:35 pm
Robert Hyatt Sat Jun 02, 2007 6:41 am
H.G.Muller Sat Jun 02, 2007 10:18 am
Robert Hyatt Sun Jun 03, 2007 12:54 am
Uri Blass Sun Jun 03, 2007 5:44 am
Robert Hyatt Sun Jun 03, 2007 8:34 pm
H.G.Muller Mon Jun 04, 2007 9:52 am
Robert Hyatt Mon Jun 04, 2007 6:58 pm
H.G.Muller Tue Jun 05, 2007 2:24 pm
Robert Hyatt Wed Jun 06, 2007 1:31 am
Uri Blass Wed Jun 06, 2007 6:42 am
Robert Hyatt Thu Jun 07, 2007 2:18 am
H.G.Muller Thu Jun 07, 2007 2:20 pm
Robert Hyatt Fri Jun 08, 2007 3:31 am
Robert Hyatt Fri Jun 08, 2007 4:02 pm
H.G.Muller Fri Jun 08, 2007 4:51 pm
Robert Hyatt Sat Jun 09, 2007 1:43 am
H.G.Muller Fri Jun 08, 2007 9:40 am
Robert Hyatt Sun Jun 10, 2007 2:24 am
Charles Roberson Wed Jun 06, 2007 2:44 am
Uri Blass Wed Jun 06, 2007 6:46 am
Ron Murawski Wed May 30, 2007 8:26 pm
Alessandro Scotti Wed May 30, 2007 8:31 pm
ed Wed May 30, 2007 11:50 pm
Dann Corbit Thu May 31, 2007 12:19 am
Dann Corbit Thu May 31, 2007 12:33 am
Dann Corbit Thu May 31, 2007 12:40 am
ed Thu May 31, 2007 9:40 am
H.G.Muller Thu May 31, 2007 11:02 am
Tony Thu May 31, 2007 12:04 pm
Re: Observator bias or... Uri Blass Thu May 31, 2007 12:51 pm
Tony Thu May 31, 2007 12:55 pm
Alessandro Scotti Thu May 31, 2007 12:56 pm
Robert Hyatt Sat Jun 02, 2007 6:37 am
Eelco de Groot Sat Jun 02, 2007 11:15 pm
Michael Sherwin Sun Jun 03, 2007 6:29 am
Uri Blass Sun Jun 03, 2007 8:11 am
Eelco de Groot Sun Jun 03, 2007 9:07 am
Uri Blass Sun Jun 03, 2007 9:39 am
H.G.Muller Sun Jun 03, 2007 9:47 am
Alessandro Scotti Sun Jun 03, 2007 8:36 am
Ron Murawski Sun Jun 03, 2007 5:50 pm
MartinBryant Sun Jun 03, 2007 9:07 am

 Jump to: Select a forum Computer Chess Club Forums----------------Computer Chess Club: General TopicsComputer Chess Club: Tournaments and MatchesComputer Chess Club: Programming and Technical DiscussionsComputer Chess Club: Engine Origins Other Forums----------------Chess Thinkers ForumForum Help and Suggestions
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum