Opening book fairness in testing

[Norm Pollock]

In order to achieve fairness in engine testing, I believe that a GUI would have to duplicate the exact same opening book position when colors are reversed.

Otherwise it can happen that A, playing white, gets a superior opening position from the the book, against B who is playing black. When the colors are reversed with B playing white, it could happen that the new opening position is neutral. In this scenario A has a distinct advantage over B in the 2 game test.

Is there a GUI out there that can remember opening book moves so that when the colors are reversed, the 2nd engine starts off in the 2nd game where the 1st engine did in the 1st game? If not, I hope to see this feature in the future.

[Dr.Wael Deeb]

The ChessBase GUI supports this feature....

[hgm]

The way people usually do this is by first generating the opening lines (e.g. by playing two book-engines against each other) and writing them to file (PGN). Then you can do symmetric testing from that file, asking the GUI or Tournament Manager to play each opening twice, with reversed colors. The advantage of this method is that you later can use the same collection of lines for other engines, that you wanted to compare to the current ones as well. (E.g. a modified version of one of the two against the same opponent.)

[BubbaTough]

In order to achieve fairness in engine testing, I believe that a GUI would have to duplicate the exact same opening book position when colors are reversed.

Otherwise it can happen that A, playing white, gets a superior opening position from the the book, against B who is playing black. When the colors are reversed with B playing white, it could happen that the new opening position is neutral. In this scenario A has a distinct advantage over B in the 2 game test.

Is there a GUI out there that can remember opening book moves so that when the colors are reversed, the 2nd engine starts off in the 2nd game where the 1st engine did in the 1st game? If not, I hope to see this feature in the future.

I do this in Arena by using the starting positions as the end of "book".

-Sam

[michiguel]

In order to achieve fairness in engine testing, I believe that a GUI would have to duplicate the exact same opening book position when colors are reversed.

Otherwise it can happen that A, playing white, gets a superior opening position from the the book, against B who is playing black. When the colors are reversed with B playing white, it could happen that the new opening position is neutral. In this scenario A has a distinct advantage over B in the 2 game test.

I believe that is a mistaken conception. What if the position favours the style and understanding of one of the opponents? The colours may be reversed but the advantage remains...

I think the most fair procedure is to have them play totally random lines in each game as long as the number of games is large enough. Then luck will even out.

I agree!

If the choice of openings is as random as possible (whatever random means in this case), reversing the colors should not be needed and should be discouraged. In fact, playing the same position with white and black may generate a non-independent result between those two games. Any statistical analysis is valid if the games are as independent as possible.

The easiest way to see this, as you mentioned, is a position that benefits the style of one engine. I think I am experiencing that with the current tests I am running with my engine Gaviota.

On the other hand, reversing the colors may not be a bad idea if you want to detect weaknesses in your engine, running always the same starting positions. With one of the nunn positions, Gaviota lost almost every single time with black and white. I found that one evaluation term was badly tuned. After removing it, this behavior was fixed for this particular position (it does not improve the overall strength, because the real solution may be to tune the parameter rather than remove it). This is just to show an extreme case of this behavior.

Miguel

[Guenther]

In order to achieve fairness in engine testing, I believe that a GUI would have to duplicate the exact same opening book position when colors are reversed.

Otherwise it can happen that A, playing white, gets a superior opening position from the the book, against B who is playing black. When the colors are reversed with B playing white, it could happen that the new opening position is neutral. In this scenario A has a distinct advantage over B in the 2 game test.

Is there a GUI out there that can remember opening book moves so that when the colors are reversed, the 2nd engine starts off in the 2nd game where the 1st engine did in the 1st game? If not, I hope to see this feature in the future.

There is still a statistical error and no fairness, if a bad opening is repeated twice, because this only helps the weaker program.
Instead of the 'normal' 0:2 or 0.5:1.5 now it gets probably a 1:1 result, just because of a repeated bad line.

Guenther

[Graham Banks]

The ChessBase GUI supports this feature....

ChessGUI also supports it.

[Spock]

The ChessBase GUI supports this feature....

ChessGUI also supports it.

And assuming I've understood correctly, so does Shredder GUI - come to think of it, is there any GUI that *doesn't* support this ?