What tests do you use for engine testing?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
royb
Posts: 520
Joined: Wed Mar 08, 2006 11:53 pm

Re: What tests do you use for engine testing?

Post by royb » Sat Jan 02, 2010 2:57 pm

David Dahlem wrote:
mhalstern wrote:Regarding opening books:

Simply have the engines play short books and make sure to play each openings (exact position out of book) and play enough games for the results to be statistically relevant.

The opening books chosen does not matter.
I disagree, choice of opening book is important. An opening book with unequal lines will give an advantage to the weaker engine.
What if the games are played such that each engine plays both sides of every opening? I think that would equalize the effect of any bad lines in the book -- am I wrong about that?

Roy

Uri Blass
Posts: 8789
Joined: Wed Mar 08, 2006 11:37 pm
Location: Tel-Aviv Israel

Re: What tests do you use for engine testing?

Post by Uri Blass » Sat Jan 02, 2010 3:03 pm

royb wrote:
David Dahlem wrote:
mhalstern wrote:Regarding opening books:

Simply have the engines play short books and make sure to play each openings (exact position out of book) and play enough games for the results to be statistically relevant.

The opening books chosen does not matter.
I disagree, choice of opening book is important. An opening book with unequal lines will give an advantage to the weaker engine.
What if the games are played such that each engine plays both sides of every opening? I think that would equalize the effect of any bad lines in the book -- am I wrong about that?

Roy
It is not going to change the fact that weak engines may score better relative to the case that you do not have bad lines.

Uri

User avatar
David Dahlem
Posts: 900
Joined: Wed Mar 08, 2006 8:06 pm

Re: What tests do you use for engine testing?

Post by David Dahlem » Sat Jan 02, 2010 4:42 pm

royb wrote:
David Dahlem wrote:
mhalstern wrote:Regarding opening books:

Simply have the engines play short books and make sure to play each openings (exact position out of book) and play enough games for the results to be statistically relevant.

The opening books chosen does not matter.
I disagree, choice of opening book is important. An opening book with unequal lines will give an advantage to the weaker engine.
What if the games are played such that each engine plays both sides of every opening? I think that would equalize the effect of any bad lines in the book -- am I wrong about that?

Roy
Yes, i think you are wrong. Unequal lines will allow a weaker engine to win or draw games that it would otherwise lose against a stronger engine if the line was equal.

Dave

mhalstern
Posts: 484
Joined: Wed Nov 18, 2009 12:09 am

Re: What tests do you use for engine testing?

Post by mhalstern » Mon Jan 04, 2010 4:53 am

David Dahlem wrote:
royb wrote:
David Dahlem wrote:
mhalstern wrote:Regarding opening books:

Simply have the engines play short books and make sure to play each openings (exact position out of book) and play enough games for the results to be statistically relevant.

The opening books chosen does not matter.
I disagree, choice of opening book is important. An opening book with unequal lines will give an advantage to the weaker engine.
What if the games are played such that each engine plays both sides of every opening? I think that would equalize the effect of any bad lines in the book -- am I wrong about that?

Roy
Yes, i think you are wrong. Unequal lines will allow a weaker engine to win or draw games that it would otherwise lose against a stronger engine if the line was equal.

Dave
I stand corrected. I was assuming that the books would be very short with equal lines.

User avatar
M ANSARI
Posts: 3466
Joined: Thu Mar 16, 2006 6:10 pm

Re: What tests do you use for engine testing?

Post by M ANSARI » Mon Jan 04, 2010 10:23 am

Actually it would also be interesting to let an engine in a winning position that is technically difficult to win, and see if it can win it. Or a very difficult draw to hold in what looks like a winning position for the opponent. Although that would have be a totally different type of test.

Post Reply