ChessUSA.com TalkChess.com
Hosted by Your Move Chess & Games
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Observator bias or...
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions Flat
View previous topic :: View next topic  
Author Message
Uri Blass



Joined: 08 Mar 2006
Posts: 5958
Location: Tel-Aviv Israel
PostPost subject: Re: Observator bias or...    Posted: Sun Jun 03, 2007 8:11 am Reply to topic Reply with quote
Michael Sherwin wrote:
Alessandro Scotti wrote:
I seem to have reached a plateau with Hamsters where new features and even bug fixes hardly contribute any elo to it.
This would bother me little if not for the fact that each and every test starts by feeding me the illusion of improvement, and only later falls to the same old stats.
A 700 games test tournament might go like this:
- games 0-100: 53% (ehi, not bad)
- games 101-200: 54% (great!)
- games 201-300: 54% (yo-hoo!... goes out to buy champagne!)
- games 301-400: 53% (just a little glitch!)
- games 401-500: 52% (some bad luck here...)
- games 501-600: 51% (you son of a...)
- games 601-700: 50% (nooooooooooooooo!!!)
So it's just my imagination or this happen to you too?!? One would think that 400 games already provide a good approximation, yet... Sad


Hi Alessandro,

I just saw your post. I have been busy giving and recieving grief in the ctf and wow was that a lot of fun. Rolling Eyes Luckily I was able to get my humour circuits powered up again for some light entertainment! Very Happy

You should have sent me an email or or a PM on this. You should know that I have played more games against Hamsters than anyone else except for possibly you. I have played many thousand RomiChess vs Hamsters games.

Yes, I have noticed this behavior with RomiChess also, but almost exclusively when playing against hamsters. Wild variation in results. One, one hundred game match of fixed positions had 56 games with different results than the match before. All I did in my code was change an 8 to a 16 for the two test.

There seems to be some random factor (?) in Hamsters that causes it to play differently a good part of the time. There are times when Romi just blows Hamsters of the board and then from the same position in the next test there seems to be nothing Romi can do against Hamsters and is herself often blown away.

Ron has also noticed this about hamsters. I am surprised that he did not mention this to you. We both have labled Hamsters as too volitile for reliable testing!

If Rybka was as random as Hamsters seems to be, then I would make the bet that Rybka would suffer a several hundred point drop in rating. Randomness limits an engines upper ceiling, no matter how many improvements are made. Randomness will lead to a certain % of losses regardless of how strong that the engine is.

This is just an educated guess on my part from playing so many games against Hamsters. I really do not know for sure. I came to the conclusion that you must have really put 5 different hamsters in your program, all with different personalities and that they are selected randomly before each game!

Unless you put some randomness into Hamsters on purpose then I do not see how it can be there. I hope that there might be something in all that I have said that might be of some help.

Mike


I disagree with you.

It is easy to change program's style by only changing the evaluation without changing the playing strength significantly.

I know it from movei and I believe that the same is for rybka and Vas can
also release some versions of rybka who play different with no big difference in playing strength(less than 40 elo difference).

Uri
Back to top
View user's profile Send private message
Display posts from previous:   
Subject Author Date/Time
Observator bias or... Alessandro Scotti Tue May 29, 2007 6:25 pm
      Re: Observator bias or... Dann Corbit Tue May 29, 2007 6:33 pm
      Re: Observator bias or... ed Tue May 29, 2007 6:59 pm
            Re: Observator bias or... H.G.Muller Wed May 30, 2007 9:43 am
                  Re: Observator bias or... ed Wed May 30, 2007 3:02 pm
                        Re: Observator bias or... cwb Wed May 30, 2007 4:48 pm
                        Re: Observator bias or... Peter Fendrich Wed May 30, 2007 6:36 pm
                        Re: Observator bias or... Robert Hyatt Sun Jun 10, 2007 3:30 pm
      Re: Observator bias or... Uri Blass Wed May 30, 2007 10:17 am
            Re: Observator bias or... Alessandro Scotti Wed May 30, 2007 12:35 pm
                  Re: Observator bias or... Robert Hyatt Sat Jun 02, 2007 6:41 am
                        Re: Observator bias or... H.G.Muller Sat Jun 02, 2007 10:18 am
                              Re: Observator bias or... Robert Hyatt Sun Jun 03, 2007 12:54 am
                                    Re: Observator bias or... Uri Blass Sun Jun 03, 2007 5:44 am
                                          Re: Observator bias or... Robert Hyatt Sun Jun 03, 2007 8:34 pm
                                                Re: Observator bias or... H.G.Muller Mon Jun 04, 2007 9:52 am
                                                      Re: Observator bias or... Robert Hyatt Mon Jun 04, 2007 6:58 pm
                                                            Re: Observator bias or... H.G.Muller Tue Jun 05, 2007 2:24 pm
                                                                  Re: Observator bias or... Robert Hyatt Wed Jun 06, 2007 1:31 am
                                                                        Re: Observator bias or... Uri Blass Wed Jun 06, 2007 6:42 am
                                                                              Re: Observator bias or... Robert Hyatt Thu Jun 07, 2007 2:18 am
                                                                        Re: Observator bias or... H.G.Muller Thu Jun 07, 2007 2:20 pm
                                                                              Re: Observator bias or... Robert Hyatt Fri Jun 08, 2007 3:31 am
                                                                                    time for some real data Robert Hyatt Fri Jun 08, 2007 4:02 pm
                                                                                          Re: time for some real data H.G.Muller Fri Jun 08, 2007 4:51 pm
                                                                                          Re: time for some real data Robert Hyatt Sat Jun 09, 2007 1:43 am
                                    Re: Observator bias or... H.G.Muller Fri Jun 08, 2007 9:40 am
                                          Re: Observator bias or... Robert Hyatt Sun Jun 10, 2007 2:24 am
                              Re: Observator bias or... Charles Roberson Wed Jun 06, 2007 2:44 am
                                    Re: Observator bias or... Uri Blass Wed Jun 06, 2007 6:46 am
      Re: Observator bias or... Ron Murawski Wed May 30, 2007 8:26 pm
            Re: Observator bias or... Alessandro Scotti Wed May 30, 2007 8:31 pm
                  Re: Observator bias or... ed Wed May 30, 2007 11:50 pm
                        Re: Observator bias or... Dann Corbit Thu May 31, 2007 12:19 am
                              Re: Observator bias or... Dann Corbit Thu May 31, 2007 12:33 am
                                    Re: Observator bias or... Dann Corbit Thu May 31, 2007 12:40 am
                                          Re: Observator bias or... ed Thu May 31, 2007 9:40 am
                  Re: Observator bias or... H.G.Muller Thu May 31, 2007 11:02 am
                        Re: Observator bias or... Tony Thu May 31, 2007 12:04 pm
                              Re: Observator bias or... Uri Blass Thu May 31, 2007 12:51 pm
                                    Re: Observator bias or... Tony Thu May 31, 2007 12:55 pm
                        Re: Observator bias or... Alessandro Scotti Thu May 31, 2007 12:56 pm
                        Re: Observator bias or... Robert Hyatt Sat Jun 02, 2007 6:37 am
                        Re: Observator bias or... Eelco de Groot Sat Jun 02, 2007 11:15 pm
      Re: Observator bias or... Michael Sherwin Sun Jun 03, 2007 6:29 am
            Re: Observator bias or... Uri Blass Sun Jun 03, 2007 8:11 am
                  Re: Observator bias or... Eelco de Groot Sun Jun 03, 2007 9:07 am
                        Re: Observator bias or... Uri Blass Sun Jun 03, 2007 9:39 am
                        Re: Observator bias or... H.G.Muller Sun Jun 03, 2007 9:47 am
            Re: Observator bias or... Alessandro Scotti Sun Jun 03, 2007 8:36 am
                  Re: Observator bias or... Ron Murawski Sun Jun 03, 2007 5:50 pm
      Re: Observator bias or... MartinBryant Sun Jun 03, 2007 9:07 am
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum




Powered by phpBB © 2001, 2005 phpBB Group
Enhanced with Moby Threads