What tests do you use for engine testing?

[M ANSARI]

I usually test engines by using a neutral book such as Harry's 8 move book or one of Sedat's Perfect books truncated at 8 or 10 moves. I heard that some are using Nunn 20 positions. Where do you get that file?

It would be great if we could choose say 100 positions that are from relevant opening theory and that would test various elements in a chess engine. Maybe some positions which test an engine's defensive abilities and offensive abilities, and then positions where positional maneuvering is required. Is there such a test set?

[swami]

I use these tests:

STS 800
Noomen Balanced 50
Noomen Gambit 50
Sedat Top 50/100/200
Nunn

Nunn positions here: http://www.hoeppenstein.net/chess/chesstutor/nunn2.fen

[istolacio]

MLmfl (Marc Lacrosse's most frequent lines)

http://sites.google.com/site/chessbazaa ... mlmfl-test

PGN: http://www.mediafire.com/?ybioiky1gmm

[alpha123]

Just head-to-head games with a relatively neutral opening book, lines not truncated . Pretty simple.

Peter

[Taner Altinsoy]

I had exactly the same issue in my mind about testing engines.
I think Nunn and Nunn2 (50 games) suite comes with Fritz package. I was wondering whether it would be a good idea to combine a few standart suites like Nunn2 and silver or any other.

I have hundreds of games played between engines in Silver suite or Nunn suite but just been hesitant about posting them since many people do not consider these games accurate for evaluating engine stentgh.

I would like to hear about this standart test suites as well. Are lets say 20, 50 or 100 mage test suite good enough for judging engine capability? What is the difference between using 100 random opening book games?

Taner Altinsoy

[M ANSARI]

Yes I agree, maybe we can come up with something like 100 or 1000 positions to test engines in. This would be the starting position as white and black for both engines and would incorporate all types of openings. It would also be really good if a sub category of performance could also be seen. Like how an engine scores in King's Indian or Sicilian for example. What is important is that these positions are as neutral as possible even if the position is dynamic.

[noctiferus]

Something alike was initiated in

http://home.interact.se/~w100107/welcome.htm

Unfortunately, it seems to have not been updated.


Moreover, there is a positional test suite...

[mhalstern]

Regarding opening books:

Simply have the engines play short books and make sure to play each openings (exact position out of book) and play enough games for the results to be statistically relevant.

The opening books chosen does not matter.

[David Dahlem]

Regarding opening books:

Simply have the engines play short books and make sure to play each openings (exact position out of book) and play enough games for the results to be statistically relevant.

The opening books chosen does not matter.

I disagree, choice of opening book is important. An opening book with unequal lines will give an advantage to the weaker engine.

[gerold]

Regarding opening books:

Simply have the engines play short books and make sure to play each openings (exact position out of book) and play enough games for the results to be statistically relevant.

The opening books chosen does not matter.

I have used some opening books that will gave the advantage
to one side when coming out of book.