TalkChess.com
Hosted by Your Move Chess & Games

Author Message
Uri Blass

Joined: 08 Mar 2006
Posts: 5958
Location: Tel-Aviv Israel

 Posted: Wed May 16, 2007 10:05 am    Post subject: a beat b,b beat c,c beat a question Do you have a case when the following happened: 1)version X+1 of your program beat version X with a significant score. 2)version X+2 of your program beat version X+1 with a significant score. 3)version X of your program beat version X+2 with a significant score. If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2? Uri
Gerd Isenberg

Joined: 08 Mar 2006
Posts: 1787
Location: Hattingen, Germany

Posted: Wed May 16, 2007 1:28 pm    Post subject: Re: a beat b,b beat c,c beat a question

 Uri Blass wrote: Do you have a case when the following happened: 1)version X+1 of your program beat version X with a significant score. 2)version X+2 of your program beat version X+1 with a significant score. 3)version X of your program beat version X+2 with a significant score. If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2? Uri

I think this is the usual szenario if you do two or more changes
The huge test-dilemma!?

Gerd
Mark Lefler

Joined: 30 Mar 2006
Posts: 500

 Posted: Wed May 16, 2007 1:41 pm    Post subject: Re: a beat b,b beat c,c beat a question The solution is simple, never, ever play version X+2 of any program versus version X and cause this loop of confusion. If you do this it will just give you a headache! I know it has me!
Michael Sherwin

Joined: 26 May 2006
Posts: 2215
Location: OH, USA

Posted: Wed May 16, 2007 1:53 pm    Post subject: Re: a beat b,b beat c,c beat a question

 Uri Blass wrote: Do you have a case when the following happened: 1)version X+1 of your program beat version X with a significant score. 2)version X+2 of your program beat version X+1 with a significant score. 3)version X of your program beat version X+2 with a significant score. If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2? Uri

This is what I call the 'marbles effect'. Take a handfull of marbles into a big flat parking lot (hope no one sees you) and then drop them a few times to see the large variance in the random patterns they create. Marbles in a programming sense is the randomness associated with any change to the code that causes one program to see what another does not or just simply causes programs to choose a different move now and then.
_________________
Regards,
Mike
Brian Richardson

Joined: 09 Mar 2006
Posts: 159

 Posted: Wed May 16, 2007 1:53 pm    Post subject: Re: a beat b,b beat c,c beat a question Firstly, I think you are already in pretty good shape if in fact you can actually test two versions of your program to the extent that you see a "significant score" difference. More importantly, many have said testing an engine against itself may highlight differences that testing an engine against a suite of other engines would not show. These same engine differences are likely to be not as meaningful as those identified by suite testing, IMO.
Christophe Théron

Joined: 11 Mar 2006
Posts: 815

Posted: Wed May 16, 2007 4:02 pm    Post subject: Re: a beat b,b beat c,c beat a question

 Uri Blass wrote: Do you have a case when the following happened: 1)version X+1 of your program beat version X with a significant score. 2)version X+2 of your program beat version X+1 with a significant score. 3)version X of your program beat version X+2 with a significant score. If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2? Uri

Even if you use 95% of confidence error margins, never forget that in one match out of 20 you will get a result outside the error bars.

How many games did you play in order to establish that "paradox"?

// Christophe
Robert Hyatt

Joined: 27 Feb 2006
Posts: 15820
Location: Birmingham, AL

Posted: Wed May 16, 2007 4:04 pm    Post subject: Re: a beat b,b beat c,c beat a question

 Uri Blass wrote: Do you have a case when the following happened: 1)version X+1 of your program beat version X with a significant score. 2)version X+2 of your program beat version X+1 with a significant score. 3)version X of your program beat version X+2 with a significant score. If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2? Uri

90% of such results are caused by playing _far_ too few games. 80 game matches mean nothing when the two programs are anywhere near close to each other.

As far as the other 10%, yes I have seen that happen, but I couldn't begin to remember what the changes were... But nothing says that this is a transitive relationship...
William H. Rogers

Joined: 09 Mar 2006
Posts: 3562
Location: San Jose, California

 Posted: Wed May 16, 2007 5:11 pm    Post subject: Re: a beat b,b beat c,c beat a question I have something similiar happen in the past and learn the following rules: 1. adding a new algorythm or new information will not cause the aboration. 2. modifing an existing algorythm in rare cases might cause the problem. 3. deleting part of all of and old algorythm can definately cause the problem. Enough of my redundant comments. Bill
Uri Blass

Joined: 08 Mar 2006
Posts: 5958
Location: Tel-Aviv Israel

Posted: Wed May 16, 2007 8:36 pm    Post subject: Re: a beat b,b beat c,c beat a question

tiger wrote:
 Uri Blass wrote: Do you have a case when the following happened: 1)version X+1 of your program beat version X with a significant score. 2)version X+2 of your program beat version X+1 with a significant score. 3)version X of your program beat version X+2 with a significant score. If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2? Uri

Even if you use 95% of confidence error margins, never forget that in one match out of 20 you will get a result outside the error bars.

How many games did you play in order to establish that "paradox"?

// Christophe

I did not claim that I played enough games to have significant results with this paradox and I ask other if it happened to them(In my case I admit that in most cases I simply did not play enough games even to get one significant result because unfortunately I am always too lazy to play thousands of games between 2 versions).

Uri
Christophe Théron

Joined: 11 Mar 2006
Posts: 815

Posted: Thu May 17, 2007 12:47 am    Post subject: Re: a beat b,b beat c,c beat a question

Uri Blass wrote:
tiger wrote:
 Uri Blass wrote: Do you have a case when the following happened: 1)version X+1 of your program beat version X with a significant score. 2)version X+2 of your program beat version X+1 with a significant score. 3)version X of your program beat version X+2 with a significant score. If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2? Uri

Even if you use 95% of confidence error margins, never forget that in one match out of 20 you will get a result outside the error bars.

How many games did you play in order to establish that "paradox"?

// Christophe

I did not claim that I played enough games to have significant results with this paradox and I ask other if it happened to them(In my case I admit that in most cases I simply did not play enough games even to get one significant result because unfortunately I am always too lazy to play thousands of games between 2 versions).

Uri

OK, then yes it happened to me even with large numbers of games, and I think you should not care too much about the kind of changes that have produced these results because they are not the real cause.

On the other hand it is probably theorically possible to design 3 programs A, B and C so A always beat B, B always beat C and C always beat A. But I'm not sure doing so will help you solve your paradox.

// Christophe
 Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First
 All times are GMT Page 1 of 1

 Jump to: Select a forum Computer Chess Club Forums----------------Computer Chess Club: General TopicsComputer Chess Club: Tournaments and MatchesComputer Chess Club: Programming and Technical DiscussionsComputer Chess Club: Engine Origins Other Forums----------------Chess Thinkers ForumForum Help and Suggestions
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum