Opening book fairness in testing

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
Norm Pollock
Posts: 1017
Joined: Thu Mar 09, 2006 3:15 pm
Location: Long Island, NY, USA
Contact:

Opening book fairness in testing

Post by Norm Pollock » Wed Jan 21, 2009 2:33 pm

In order to achieve fairness in engine testing, I believe that a GUI would have to duplicate the exact same opening book position when colors are reversed.

Otherwise it can happen that A, playing white, gets a superior opening position from the the book, against B who is playing black. When the colors are reversed with B playing white, it could happen that the new opening position is neutral. In this scenario A has a distinct advantage over B in the 2 game test.

Is there a GUI out there that can remember opening book moves so that when the colors are reversed, the 2nd engine starts off in the 2nd game where the 1st engine did in the 1st game? If not, I hope to see this feature in the future.

User avatar
Dr.Wael Deeb
Posts: 9635
Joined: Wed Mar 08, 2006 7:44 pm
Location: Amman,Jordan

Re: Opening book fairness in testing

Post by Dr.Wael Deeb » Wed Jan 21, 2009 2:58 pm

The ChessBase GUI supports this feature....
_No one can hit as hard as life.But it ain’t about how hard you can hit.It’s about how hard you can get hit and keep moving forward.How much you can take and keep moving forward….

User avatar
hgm
Posts: 23785
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: Opening book fairness in testing

Post by hgm » Wed Jan 21, 2009 3:04 pm

The way people usually do this is by first generating the opening lines (e.g. by playing two book-engines against each other) and writing them to file (PGN). Then you can do symmetric testing from that file, asking the GUI or Tournament Manager to play each opening twice, with reversed colors. The advantage of this method is that you later can use the same collection of lines for other engines, that you wanted to compare to the current ones as well. (E.g. a modified version of one of the two against the same opponent.)

BubbaTough
Posts: 1154
Joined: Fri Jun 23, 2006 3:18 am

Re: Opening book fairness in testing

Post by BubbaTough » Wed Jan 21, 2009 3:06 pm

Norm Pollock wrote:In order to achieve fairness in engine testing, I believe that a GUI would have to duplicate the exact same opening book position when colors are reversed.

Otherwise it can happen that A, playing white, gets a superior opening position from the the book, against B who is playing black. When the colors are reversed with B playing white, it could happen that the new opening position is neutral. In this scenario A has a distinct advantage over B in the 2 game test.

Is there a GUI out there that can remember opening book moves so that when the colors are reversed, the 2nd engine starts off in the 2nd game where the 1st engine did in the 1st game? If not, I hope to see this feature in the future.
I do this in Arena by using the starting positions as the end of "book".

-Sam

User avatar
michiguel
Posts: 6389
Joined: Thu Mar 09, 2006 7:30 pm
Location: Chicago, Illinois, USA
Contact:

Re: Opening book fairness in testing

Post by michiguel » Wed Jan 21, 2009 5:28 pm

SzG wrote:
Norm Pollock wrote:In order to achieve fairness in engine testing, I believe that a GUI would have to duplicate the exact same opening book position when colors are reversed.

Otherwise it can happen that A, playing white, gets a superior opening position from the the book, against B who is playing black. When the colors are reversed with B playing white, it could happen that the new opening position is neutral. In this scenario A has a distinct advantage over B in the 2 game test.
I believe that is a mistaken conception. What if the position favours the style and understanding of one of the opponents? The colours may be reversed but the advantage remains...

I think the most fair procedure is to have them play totally random lines in each game as long as the number of games is large enough. Then luck will even out.
I agree!

If the choice of openings is as random as possible (whatever random means in this case), reversing the colors should not be needed and should be discouraged. In fact, playing the same position with white and black may generate a non-independent result between those two games. Any statistical analysis is valid if the games are as independent as possible.

The easiest way to see this, as you mentioned, is a position that benefits the style of one engine. I think I am experiencing that with the current tests I am running with my engine Gaviota.

On the other hand, reversing the colors may not be a bad idea if you want to detect weaknesses in your engine, running always the same starting positions. With one of the nunn positions, Gaviota lost almost every single time with black and white. I found that one evaluation term was badly tuned. After removing it, this behavior was fixed for this particular position (it does not improve the overall strength, because the real solution may be to tune the parameter rather than remove it). This is just to show an extreme case of this behavior.

Miguel

User avatar
Guenther
Posts: 3111
Joined: Wed Oct 01, 2008 4:33 am
Location: Regensburg, Germany
Full name: Guenther Simon
Contact:

Re: Opening book fairness in testing

Post by Guenther » Wed Jan 21, 2009 6:02 pm

Norm Pollock wrote:In order to achieve fairness in engine testing, I believe that a GUI would have to duplicate the exact same opening book position when colors are reversed.

Otherwise it can happen that A, playing white, gets a superior opening position from the the book, against B who is playing black. When the colors are reversed with B playing white, it could happen that the new opening position is neutral. In this scenario A has a distinct advantage over B in the 2 game test.

Is there a GUI out there that can remember opening book moves so that when the colors are reversed, the 2nd engine starts off in the 2nd game where the 1st engine did in the 1st game? If not, I hope to see this feature in the future.
There is still a statistical error and no fairness, if a bad opening is repeated twice, because this only helps the weaker program.
Instead of the 'normal' 0:2 or 0.5:1.5 now it gets probably a 1:1 result, just because of a repeated bad line.

Guenther

User avatar
Graham Banks
Posts: 33245
Joined: Sun Feb 26, 2006 9:52 am
Location: Auckland, NZ

Re: Opening book fairness in testing

Post by Graham Banks » Wed Jan 21, 2009 6:29 pm

Dr.Wael Deeb wrote:The ChessBase GUI supports this feature....
ChessGUI also supports it. 8-)
My email addresses:
gbanksnz at gmail.com
gbanksnz at yahoo.co.nz

Spock

Re: Opening book fairness in testing

Post by Spock » Wed Jan 21, 2009 9:39 pm

Graham Banks wrote:
Dr.Wael Deeb wrote:The ChessBase GUI supports this feature....
ChessGUI also supports it. 8-)
And assuming I've understood correctly, so does Shredder GUI - come to think of it, is there any GUI that *doesn't* support this ?

Post Reply