Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.
If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?
Uri
a beat b,b beat c,c beat a question
Moderator: Ras
-
- Posts: 10693
- Joined: Thu Mar 09, 2006 12:37 am
- Location: Tel-Aviv Israel
-
- Posts: 2251
- Joined: Wed Mar 08, 2006 8:47 pm
- Location: Hattingen, Germany
Re: a beat b,b beat c,c beat a question
I think this is the usual szenario if you do two or more changesUri Blass wrote:Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.
If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?
Uri

The huge test-dilemma!?
Gerd
-
- Posts: 1494
- Joined: Thu Mar 30, 2006 2:08 pm
Re: a beat b,b beat c,c beat a question
The solution is simple, never, ever play version X+2 of any program versus version X and cause this loop of confusion. If you do this it will just give you a headache! I know it has me!
-
- Posts: 3196
- Joined: Fri May 26, 2006 3:00 am
- Location: WY, USA
- Full name: Michael Sherwin
Re: a beat b,b beat c,c beat a question
This is what I call the 'marbles effect'. Take a handfull of marbles into a big flat parking lot (hope no one sees you) and then drop them a few times to see the large variance in the random patterns they create. Marbles in a programming sense is the randomness associated with any change to the code that causes one program to see what another does not or just simply causes programs to choose a different move now and then.Uri Blass wrote:Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.
If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?
Uri
If you are on a sidewalk and the covid goes beep beep
Just step aside or you might have a bit of heat
Covid covid runs through the town all day
Can the people ever change their ways
Sherwin the covid's after you
Sherwin if it catches you you're through
Just step aside or you might have a bit of heat
Covid covid runs through the town all day
Can the people ever change their ways
Sherwin the covid's after you
Sherwin if it catches you you're through
-
- Posts: 540
- Joined: Thu Mar 09, 2006 3:01 pm
- Full name: Brian Richardson
Re: a beat b,b beat c,c beat a question
Firstly, I think you are already in pretty good shape if in fact you can actually test two versions of your program to the extent that you see a "significant score" difference.
More importantly, many have said testing an engine against itself may highlight differences that testing an engine against a suite of other engines would not show. These same engine differences are likely to be not as meaningful as those identified by suite testing, IMO.
More importantly, many have said testing an engine against itself may highlight differences that testing an engine against a suite of other engines would not show. These same engine differences are likely to be not as meaningful as those identified by suite testing, IMO.
-
- Posts: 819
- Joined: Sat Mar 11, 2006 3:15 am
- Location: Guadeloupe (french caribbean island)
Re: a beat b,b beat c,c beat a question
Uri Blass wrote:Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.
If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?
Uri
The answer to this paradox is probably in your definition of "beat with a significant score".
Even if you use 95% of confidence error margins, never forget that in one match out of 20 you will get a result outside the error bars.
How many games did you play in order to establish that "paradox"?
// Christophe
-
- Posts: 20943
- Joined: Mon Feb 27, 2006 7:30 pm
- Location: Birmingham, AL
Re: a beat b,b beat c,c beat a question
90% of such results are caused by playing _far_ too few games. 80 game matches mean nothing when the two programs are anywhere near close to each other.Uri Blass wrote:Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.
If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?
Uri
As far as the other 10%, yes I have seen that happen, but I couldn't begin to remember what the changes were... But nothing says that this is a transitive relationship...
-
- Posts: 3562
- Joined: Thu Mar 09, 2006 3:54 am
- Location: San Jose, California
Re: a beat b,b beat c,c beat a question
I have something similiar happen in the past and learn the following rules:
1. adding a new algorythm or new information will not cause the aboration.
2. modifing an existing algorythm in rare cases might cause the problem.
3. deleting part of all of and old algorythm can definately cause the problem.
Enough of my redundant comments.
Bill
1. adding a new algorythm or new information will not cause the aboration.
2. modifing an existing algorythm in rare cases might cause the problem.
3. deleting part of all of and old algorythm can definately cause the problem.
Enough of my redundant comments.
Bill
-
- Posts: 10693
- Joined: Thu Mar 09, 2006 12:37 am
- Location: Tel-Aviv Israel
Re: a beat b,b beat c,c beat a question
I did not claim that I played enough games to have significant results with this paradox and I ask other if it happened to them(In my case I admit that in most cases I simply did not play enough games even to get one significant result because unfortunately I am always too lazy to play thousands of games between 2 versions).tiger wrote:Uri Blass wrote:Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.
If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?
Uri
The answer to this paradox is probably in your definition of "beat with a significant score".
Even if you use 95% of confidence error margins, never forget that in one match out of 20 you will get a result outside the error bars.
How many games did you play in order to establish that "paradox"?
// Christophe
Uri
-
- Posts: 819
- Joined: Sat Mar 11, 2006 3:15 am
- Location: Guadeloupe (french caribbean island)
Re: a beat b,b beat c,c beat a question
Uri Blass wrote:I did not claim that I played enough games to have significant results with this paradox and I ask other if it happened to them(In my case I admit that in most cases I simply did not play enough games even to get one significant result because unfortunately I am always too lazy to play thousands of games between 2 versions).tiger wrote:Uri Blass wrote:Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.
If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?
Uri
The answer to this paradox is probably in your definition of "beat with a significant score".
Even if you use 95% of confidence error margins, never forget that in one match out of 20 you will get a result outside the error bars.
How many games did you play in order to establish that "paradox"?
// Christophe
Uri
OK, then yes it happened to me even with large numbers of games, and I think you should not care too much about the kind of changes that have produced these results because they are not the real cause.
On the other hand it is probably theorically possible to design 3 programs A, B and C so A always beat B, B always beat C and C always beat A. But I'm not sure doing so will help you solve your paradox.
// Christophe