a beat b,b beat c,c beat a question

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
Uri Blass
Posts: 8436
Joined: Wed Mar 08, 2006 11:37 pm
Location: Tel-Aviv Israel

a beat b,b beat c,c beat a question

Post by Uri Blass » Wed May 16, 2007 10:05 am

Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.

If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?

Uri

Gerd Isenberg
Posts: 2125
Joined: Wed Mar 08, 2006 7:47 pm
Location: Hattingen, Germany

Re: a beat b,b beat c,c beat a question

Post by Gerd Isenberg » Wed May 16, 2007 1:28 pm

Uri Blass wrote:Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.

If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?

Uri
I think this is the usual szenario if you do two or more changes ;-)
The huge test-dilemma!?

Gerd

mjlef
Posts: 1402
Joined: Thu Mar 30, 2006 12:08 pm
Contact:

Re: a beat b,b beat c,c beat a question

Post by mjlef » Wed May 16, 2007 1:41 pm

The solution is simple, never, ever play version X+2 of any program versus version X and cause this loop of confusion. If you do this it will just give you a headache! I know it has me!

Michael Sherwin
Posts: 3024
Joined: Fri May 26, 2006 1:00 am
Location: WY, USA
Full name: Michael Sherwin

Re: a beat b,b beat c,c beat a question

Post by Michael Sherwin » Wed May 16, 2007 1:53 pm

Uri Blass wrote:Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.

If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?

Uri
This is what I call the 'marbles effect'. Take a handfull of marbles into a big flat parking lot (hope no one sees you) and then drop them a few times to see the large variance in the random patterns they create. Marbles in a programming sense is the randomness associated with any change to the code that causes one program to see what another does not or just simply causes programs to choose a different move now and then.
I hate if statements. Pawns demand if statements. Therefore I hate pawns.

brianr
Posts: 343
Joined: Thu Mar 09, 2006 2:01 pm

Re: a beat b,b beat c,c beat a question

Post by brianr » Wed May 16, 2007 1:53 pm

Firstly, I think you are already in pretty good shape if in fact you can actually test two versions of your program to the extent that you see a "significant score" difference.

More importantly, many have said testing an engine against itself may highlight differences that testing an engine against a suite of other engines would not show. These same engine differences are likely to be not as meaningful as those identified by suite testing, IMO.

User avatar
tiger
Posts: 819
Joined: Sat Mar 11, 2006 2:15 am
Location: Guadeloupe (french caribbean island)
Contact:

Re: a beat b,b beat c,c beat a question

Post by tiger » Wed May 16, 2007 4:02 pm

Uri Blass wrote:Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.

If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?

Uri

The answer to this paradox is probably in your definition of "beat with a significant score".

Even if you use 95% of confidence error margins, never forget that in one match out of 20 you will get a result outside the error bars.

How many games did you play in order to establish that "paradox"?


// Christophe

bob
Posts: 20373
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: a beat b,b beat c,c beat a question

Post by bob » Wed May 16, 2007 4:04 pm

Uri Blass wrote:Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.

If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?

Uri
90% of such results are caused by playing _far_ too few games. 80 game matches mean nothing when the two programs are anywhere near close to each other.

As far as the other 10%, yes I have seen that happen, but I couldn't begin to remember what the changes were... But nothing says that this is a transitive relationship...

User avatar
Bill Rogers
Posts: 3562
Joined: Thu Mar 09, 2006 2:54 am
Location: San Jose, California

Re: a beat b,b beat c,c beat a question

Post by Bill Rogers » Wed May 16, 2007 5:11 pm

I have something similiar happen in the past and learn the following rules:
1. adding a new algorythm or new information will not cause the aboration.
2. modifing an existing algorythm in rare cases might cause the problem.
3. deleting part of all of and old algorythm can definately cause the problem.
Enough of my redundant comments.
Bill

Uri Blass
Posts: 8436
Joined: Wed Mar 08, 2006 11:37 pm
Location: Tel-Aviv Israel

Re: a beat b,b beat c,c beat a question

Post by Uri Blass » Wed May 16, 2007 8:36 pm

tiger wrote:
Uri Blass wrote:Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.

If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?

Uri

The answer to this paradox is probably in your definition of "beat with a significant score".

Even if you use 95% of confidence error margins, never forget that in one match out of 20 you will get a result outside the error bars.

How many games did you play in order to establish that "paradox"?


// Christophe
I did not claim that I played enough games to have significant results with this paradox and I ask other if it happened to them(In my case I admit that in most cases I simply did not play enough games even to get one significant result because unfortunately I am always too lazy to play thousands of games between 2 versions).

Uri

User avatar
tiger
Posts: 819
Joined: Sat Mar 11, 2006 2:15 am
Location: Guadeloupe (french caribbean island)
Contact:

Re: a beat b,b beat c,c beat a question

Post by tiger » Thu May 17, 2007 12:47 am

Uri Blass wrote:
tiger wrote:
Uri Blass wrote:Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.

If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?

Uri

The answer to this paradox is probably in your definition of "beat with a significant score".

Even if you use 95% of confidence error margins, never forget that in one match out of 20 you will get a result outside the error bars.

How many games did you play in order to establish that "paradox"?


// Christophe
I did not claim that I played enough games to have significant results with this paradox and I ask other if it happened to them(In my case I admit that in most cases I simply did not play enough games even to get one significant result because unfortunately I am always too lazy to play thousands of games between 2 versions).

Uri

OK, then yes it happened to me even with large numbers of games, and I think you should not care too much about the kind of changes that have produced these results because they are not the real cause.

On the other hand it is probably theorically possible to design 3 programs A, B and C so A always beat B, B always beat C and C always beat A. But I'm not sure doing so will help you solve your paradox.


// Christophe

Post Reply