What tests do you use for engine testing?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
M ANSARI
Posts: 3466
Joined: Thu Mar 16, 2006 6:10 pm

What tests do you use for engine testing?

Post by M ANSARI » Fri Jan 01, 2010 5:27 am

I usually test engines by using a neutral book such as Harry's 8 move book or one of Sedat's Perfect books truncated at 8 or 10 moves. I heard that some are using Nunn 20 positions. Where do you get that file?

It would be great if we could choose say 100 positions that are from relevant opening theory and that would test various elements in a chess engine. Maybe some positions which test an engine's defensive abilities and offensive abilities, and then positions where positional maneuvering is required. Is there such a test set?

swami
Posts: 6546
Joined: Thu Mar 09, 2006 3:21 am

Re: What tests do you use for engine testing?

Post by swami » Fri Jan 01, 2010 5:45 am

I use these tests:

STS 800
Noomen Balanced 50
Noomen Gambit 50
Sedat Top 50/100/200
Nunn

Nunn positions here: http://www.hoeppenstein.net/chess/chesstutor/nunn2.fen

istolacio
Posts: 10
Joined: Wed Aug 30, 2006 10:29 pm
Location: Valencia, Spain

Re: What tests do you use for engine testing?

Post by istolacio » Fri Jan 01, 2010 6:52 am


alpha123
Posts: 660
Joined: Sat Dec 05, 2009 4:13 am
Location: Colorado, USA

Re: What tests do you use for engine testing?

Post by alpha123 » Fri Jan 01, 2010 7:33 am

Just head-to-head games with a relatively neutral opening book, lines not truncated :D. Pretty simple.

Peter

Taner Altinsoy
Posts: 147
Joined: Fri Dec 18, 2009 2:56 pm
Location: Istanbul

Re: What tests do you use for engine testing?

Post by Taner Altinsoy » Fri Jan 01, 2010 12:19 pm

I had exactly the same issue in my mind about testing engines.
I think Nunn and Nunn2 (50 games) suite comes with Fritz package. I was wondering whether it would be a good idea to combine a few standart suites like Nunn2 and silver or any other.

I have hundreds of games played between engines in Silver suite or Nunn suite but just been hesitant about posting them since many people do not consider these games accurate for evaluating engine stentgh.

I would like to hear about this standart test suites as well. Are lets say 20, 50 or 100 mage test suite good enough for judging engine capability? What is the difference between using 100 random opening book games?

Taner Altinsoy

User avatar
M ANSARI
Posts: 3466
Joined: Thu Mar 16, 2006 6:10 pm

Re: What tests do you use for engine testing?

Post by M ANSARI » Fri Jan 01, 2010 1:37 pm

Yes I agree, maybe we can come up with something like 100 or 1000 positions to test engines in. This would be the starting position as white and black for both engines and would incorporate all types of openings. It would also be really good if a sub category of performance could also be seen. Like how an engine scores in King's Indian or Sicilian for example. What is important is that these positions are as neutral as possible even if the position is dynamic.

noctiferus
Posts: 364
Joined: Sun Oct 04, 2009 11:27 am
Location: Italy

Re: What tests do you use for engine testing?

Post by noctiferus » Fri Jan 01, 2010 1:59 pm

Something alike was initiated in

http://home.interact.se/~w100107/welcome.htm

Unfortunately, it seems to have not been updated.


Moreover, there is a positional test suite...

mhalstern
Posts: 484
Joined: Wed Nov 18, 2009 12:09 am

Re: What tests do you use for engine testing?

Post by mhalstern » Sat Jan 02, 2010 12:50 am

Regarding opening books:

Simply have the engines play short books and make sure to play each openings (exact position out of book) and play enough games for the results to be statistically relevant.

The opening books chosen does not matter.

User avatar
David Dahlem
Posts: 900
Joined: Wed Mar 08, 2006 8:06 pm

Re: What tests do you use for engine testing?

Post by David Dahlem » Sat Jan 02, 2010 1:57 pm

mhalstern wrote:Regarding opening books:

Simply have the engines play short books and make sure to play each openings (exact position out of book) and play enough games for the results to be statistically relevant.

The opening books chosen does not matter.
I disagree, choice of opening book is important. An opening book with unequal lines will give an advantage to the weaker engine.

gerold
Posts: 10121
Joined: Wed Mar 08, 2006 11:57 pm
Location: van buren,missouri

Re: What tests do you use for engine testing?

Post by gerold » Sat Jan 02, 2010 2:38 pm

mhalstern wrote:Regarding opening books:

Simply have the engines play short books and make sure to play each openings (exact position out of book) and play enough games for the results to be statistically relevant.

The opening books chosen does not matter.
I have used some opening books that will gave the advantage
to one side when coming out of book.

Post Reply