| View previous topic :: View next topic |
| Author |
Message |
Uri Blass
Joined: 08 Mar 2006
Posts: 5958
Location: Tel-Aviv Israel
|
 Posted: Wed May 16, 2007 10:05 am Post subject: a beat b,b beat c,c beat a question |
  |
|
Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.
If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?
Uri |
|
| Back to top |
|
 |
Gerd Isenberg
Joined: 08 Mar 2006
Posts: 1787
Location: Hattingen, Germany
|
| Posted: Wed May 16, 2007 1:28 pm Post subject: Re: a beat b,b beat c,c beat a question |
|
|
| Uri Blass wrote: |
Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.
If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?
Uri |
I think this is the usual szenario if you do two or more changes 
The huge test-dilemma!?
Gerd |
|
| Back to top |
|
|
Mark Lefler
Joined: 30 Mar 2006
Posts: 500
|
| Posted: Wed May 16, 2007 1:41 pm Post subject: Re: a beat b,b beat c,c beat a question |
|
|
| The solution is simple, never, ever play version X+2 of any program versus version X and cause this loop of confusion. If you do this it will just give you a headache! I know it has me! |
|
| Back to top |
|
|
Michael Sherwin
Joined: 26 May 2006
Posts: 2215
Location: OH, USA
|
| Posted: Wed May 16, 2007 1:53 pm Post subject: Re: a beat b,b beat c,c beat a question |
|
|
| Uri Blass wrote: |
Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.
If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?
Uri |
This is what I call the 'marbles effect'. Take a handfull of marbles into a big flat parking lot (hope no one sees you) and then drop them a few times to see the large variance in the random patterns they create. Marbles in a programming sense is the randomness associated with any change to the code that causes one program to see what another does not or just simply causes programs to choose a different move now and then.
_________________
Regards,
Mike |
|
| Back to top |
|
|
Brian Richardson
Joined: 09 Mar 2006
Posts: 159
|
| Posted: Wed May 16, 2007 1:53 pm Post subject: Re: a beat b,b beat c,c beat a question |
|
|
Firstly, I think you are already in pretty good shape if in fact you can actually test two versions of your program to the extent that you see a "significant score" difference.
More importantly, many have said testing an engine against itself may highlight differences that testing an engine against a suite of other engines would not show. These same engine differences are likely to be not as meaningful as those identified by suite testing, IMO. |
|
| Back to top |
|
|
Christophe Théron
Joined: 11 Mar 2006
Posts: 815
Location: Guadeloupe (french caribbean island)
|
| Posted: Wed May 16, 2007 4:02 pm Post subject: Re: a beat b,b beat c,c beat a question |
|
|
| Uri Blass wrote: |
Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.
If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?
Uri |
The answer to this paradox is probably in your definition of "beat with a significant score".
Even if you use 95% of confidence error margins, never forget that in one match out of 20 you will get a result outside the error bars.
How many games did you play in order to establish that "paradox"?
// Christophe |
|
| Back to top |
|
|
Robert Hyatt
Joined: 27 Feb 2006
Posts: 15820
Location: Birmingham, AL
|
| Posted: Wed May 16, 2007 4:04 pm Post subject: Re: a beat b,b beat c,c beat a question |
|
|
| Uri Blass wrote: |
Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.
If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?
Uri |
90% of such results are caused by playing _far_ too few games. 80 game matches mean nothing when the two programs are anywhere near close to each other.
As far as the other 10%, yes I have seen that happen, but I couldn't begin to remember what the changes were... But nothing says that this is a transitive relationship... |
|
| Back to top |
|
|
William H. Rogers
Joined: 09 Mar 2006
Posts: 3562
Location: San Jose, California
|
| Posted: Wed May 16, 2007 5:11 pm Post subject: Re: a beat b,b beat c,c beat a question |
|
|
I have something similiar happen in the past and learn the following rules:
1. adding a new algorythm or new information will not cause the aboration.
2. modifing an existing algorythm in rare cases might cause the problem.
3. deleting part of all of and old algorythm can definately cause the problem.
Enough of my redundant comments.
Bill |
|
| Back to top |
|
|
Uri Blass
Joined: 08 Mar 2006
Posts: 5958
Location: Tel-Aviv Israel
|
| Posted: Wed May 16, 2007 8:36 pm Post subject: Re: a beat b,b beat c,c beat a question |
|
|
| tiger wrote: |
| Uri Blass wrote: |
Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.
If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?
Uri |
The answer to this paradox is probably in your definition of "beat with a significant score".
Even if you use 95% of confidence error margins, never forget that in one match out of 20 you will get a result outside the error bars.
How many games did you play in order to establish that "paradox"?
// Christophe |
I did not claim that I played enough games to have significant results with this paradox and I ask other if it happened to them(In my case I admit that in most cases I simply did not play enough games even to get one significant result because unfortunately I am always too lazy to play thousands of games between 2 versions).
Uri |
|
| Back to top |
|
|
Christophe Théron
Joined: 11 Mar 2006
Posts: 815
Location: Guadeloupe (french caribbean island)
|
| Posted: Thu May 17, 2007 12:47 am Post subject: Re: a beat b,b beat c,c beat a question |
|
|
| Uri Blass wrote: |
| tiger wrote: |
| Uri Blass wrote: |
Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.
If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?
Uri |
The answer to this paradox is probably in your definition of "beat with a significant score".
Even if you use 95% of confidence error margins, never forget that in one match out of 20 you will get a result outside the error bars.
How many games did you play in order to establish that "paradox"?
// Christophe |
I did not claim that I played enough games to have significant results with this paradox and I ask other if it happened to them(In my case I admit that in most cases I simply did not play enough games even to get one significant result because unfortunately I am always too lazy to play thousands of games between 2 versions).
Uri |
OK, then yes it happened to me even with large numbers of games, and I think you should not care too much about the kind of changes that have produced these results because they are not the real cause.
On the other hand it is probably theorically possible to design 3 programs A, B and C so A always beat B, B always beat C and C always beat A. But I'm not sure doing so will help you solve your paradox.
// Christophe |
|
| Back to top |
|
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
