'STS' Test Suite - Available for Testing

[Jouni]

Google to epd2pgn converter. BTW I run suite with more engines and believe or not: it gave exactly correct strength order for engines

Jouni

[Cubeman]

Thanks for the converter tool.I managed to get the positions converted to PGN but it only gives the start position with no moves, so it is difficult to know what the actual correct move is.

[Dann Corbit]

Finally new testsuite, thanks! But isn't this TOO EASY: Naum and Grapefruit (1 CPU) got 90-91/100 with one minute limit only.

Jouni

It could be easy for really stronger engines but do not forget that: Main objective is that it's designed and is more suitable to engines ranging from 1500-2700. Rybka, Naum and Fruit can solve many.

I'm surprised that they solved only 90/100 with 1 minute limit. They would probably have solved 100/100 in tactics. Anyway Thanks for posting the results.

I see the test suite having several uses.
First, we can run it at very high speed (e.g. 1 sec/pos) to see if the main idea is improved by a source change.
Second, we can run it at (perhaps) 1 minute per position to compare various engines as far as number solved and time to solution.
Third, we can use it as human readable format to try to solve the positions ourselves.
Fourth, the zillion other things I did not think of yet.

[Dann Corbit]

What tool is it that you are using that cannot even read EPD?

Tools that can read EPD include:
ChessAssistant
ChessBase
SCID
Bookup
Aquarium
Arena

[David Dahlem]

In case anyone is interested, i decided to try and convert this test suite to a format that can be used by the Gradual Test utility by Odd Gunnar Malin. When i got to this position, i noticed that the move "Qc8" was listed twice, with different scores ...

1q2r1k1/1b2bpp1/p2ppn1p/2p5/P3PP1B/2PB1RP1/2P1Q2P/2KR4 b - - bm c4; id "Undermine.005"; c0 "c4=10, Bc6=7, Qa8=7, Qc8=6, Qc8=7";

Most likely it's a typo and one of those "Qc8" moves should be "Qd8". In any case, what should be the correct moves and scores?



And if i manage to finish the conversion, i'll post it here if anyone is interested.

Regards
Dave

[David Dahlem]

Here's another one. "Nf1" is listed twice, with different scores....

1r6/R1nk1p2/1p4pp/pP1p1P2/P2P3P/5PN1/5K2/8 w - - bm h5; id "Undermine.019"; c0 "h5=10, Ne2=4, Nf1=6, Nf1=7, f4=7";

Regards
Dave

[Will Singleton]

position 3 might be busted, try rd5

[Spock]

Finally new testsuite, thanks! But isn't this TOO EASY: Naum and Grapefruit (1 CPU) got 90-91/100 with one minute limit only.

Jouni

It could be easy for really stronger engines but do not forget that: Main objective is that it's designed and is more suitable to engines ranging from 1500-2700. Rybka, Naum and Fruit can solve many.

I'm surprised that they solved only 90/100 with 1 minute limit. They would probably have solved 100/100 in tactics. Anyway Thanks for posting the results.

I tried Rotor 0.4 at the suggested 7 mins per move. This is an approx 2600 rated engine on CCRL 40/40. It found 79 of 100 so it is definitely tougher for the slighly lower rated engines. It would be interesting to try something further down

[swami]

Thanks for the bug reports. I will ask Dann Corbitt to release the newer one sooner with corrections - since most of the errors are related to wrong points scores/duplicate moves.

[swami]

Google to epd2pgn converter. BTW I run suite with more engines and believe or not: it gave exactly correct strength order for engines

Jouni

That's indeed good to know!
May I know the list of engines you tested and their scores?