Question to all testers

[Jim Walker]

If you were running your favorite program in a tournament for prize money would you be putting all the silly limitations on the program that you do in your testing? Such as limited books, ponder off, no tablebases,no position learning, no book learning etc.etc. If not then why all the interest in testing programs under conditions which they were not designed for? Just curious.
Jim

[Graham Banks]

If you were running your favorite program in a tournament for prize money would you be putting all the silly limitations on the program that you do in your testing? Such as limited books, ponder off, no tablebases,no position learning, no book learning etc.etc. If not then why all the interest in testing programs under conditions which they were not designed for? Just curious.
Jim

You paint two entirely different scenarios Jim.

In a tournamenr, you'd be looking for every little advantage you could get for your engine in order to win that prize.

In CCRL testing, we're attempting to test engine strength without the influence of their own books or learning.
Each of these factors gives some engines an advantage over others as you're aware.
That's why we use a generic book for all engines in any matches or tournaments that we run.

Testing with ponder on or ponder off makes little difference to the relative ratings of engines.
Such comparisons have been done several times over the years.
Also, testers or users who have single-cpu computers don't have the ponder on option available to them.

Both CCRL and CEGT use generic books, learning off and ponder off for their testing.
SSDF do not.
It's all a matter of preference.

Regards, Graham.

[Rubinus]

... Both CCRL and CEGT use generic books, learning off and ponder off for their testing. ....

Very good for new openings theory, thanks. But, if own books use, more interesting games.

Best wishes Pavel Háse

[Jim Walker]

Thanks for the response.

[Tony Thomas]

I only know of one tournament that plays ponder on, own book, learning on etc..WBEC.

[Mike S.]

Ponder on or off will usually depend on the tester's hardware. With 2 comps or dual core, I assume all testers would prefer to test single cpu engines with ponder=on, against each other. They may of course still test with ponder=off if they submit results to a tester group which has this condition.

Books and -learning are a kind of philosophy question, IOW what is the normal situation in practise, for which the test result should provide information? Like, you won't test a sports car offroad and you won't test a Jeep in a racing oval. If it is top-level engine sport like the Freestyle tournaments or computer ratings competition, it makes sense to test with individual books per engine with best tuning and book learning.

But that is a very small sector of computerchess practise. For the normal chess player, it's not important in which opening an engine performs best, because for him his own repertoire and/or the opening he explores etc. are important, not what the engine "likes." The engine has to support him in the openings HE likes. That's why for example I use a neutral medium-deep test book (Xmas2640) for the Blivorix tests, which was designed to provide a better variety (wide repertoire). This is much unlike the usual engine books or simple game-based books, where i.e. the Sicilian and Ruy Lopez get a much too big share of all openings played.

http://members.aon.at/computerschach/li ... #downloads

There are two versions, for Fritz and for Arena. It mainly is a test book, NOT a performance book. Some openings which are rare in GM practise, like KG. or Philidor, are not included.

As for position learning and for tablebases, I have set both to on basically, with the exception that I don't use a full 3/4/5-piece tbs. set but a smaller subset.

Only if I use test suites (positions), I delete the learn files before, to ensure "clean" test conditions. Because often, some engines will have analysed in some of the positions before and may have stored something. One could also temporarily move the learn files to another directory instead of deleting them, to keep the learning data.