Tournament readiness

[stevemulligan]

I'm wondering what features my engine needs to be tournament ready for normal chess in the average tournament.

A recent thread in the tournament forums has some items that I don't understand like:

128mb hash each
3-4-5 piece tablebases
Charlotte 1.0 mini.cgb book


Also, before I start tuning my engine I want to get a baseline ELO. What is the best way to go about that?

[jshriver]

I would just have it play a bunch of games against other engines and make a pgn database. Then feed that into bayselo program to get a elo rating.

Congrats on the engine
-Josh

[JVMerlino]

I'm wondering what features my engine needs to be tournament ready for normal chess in the average tournament.

A recent thread in the tournament forums has some items that I don't understand like:

128mb hash each
3-4-5 piece tablebases
Charlotte 1.0 mini.cgb book

Also, before I start tuning my engine I want to get a baseline ELO. What is the best way to go about that?

For "tournament readiness", IMHO the most important thing you need is stability under whatever communication protocol, GUI and time control the tournament director is going to be using. And the last of these is probably the most important. I've seen dozens of games lost by engines that didn't properly support the time control, and many others by engines that just crashed.

Strength will eventually come when you implement hash tables, tablebases, opening books, and other typical improvements. But without a stable engine, it is impossible to truly determine how strong your engine is.

Just my two cents....

jm

[JVMerlino]

I'm wondering what features my engine needs to be tournament ready for normal chess in the average tournament.

A recent thread in the tournament forums has some items that I don't understand like:

128mb hash each
3-4-5 piece tablebases
Charlotte 1.0 mini.cgb book

Also, before I start tuning my engine I want to get a baseline ELO. What is the best way to go about that?

For "tournament readiness", IMHO the most important thing you need is stability under whatever communication protocol, GUI and time control the tournament director is going to be using. And the last of these is probably the most important. I've seen dozens of games lost by engines that didn't properly support the time control, and many others by engines that just crashed.

Strength will eventually come when you implement hash tables, tablebases, opening books, and other typical improvements. But without a stable engine, it is impossible to truly determine how strong your engine is.

Just my two cents....

jm

Of course, in my haste to give my two cents, I didn't consider the possibility that you already have a stable engine that supports all time controls. If you do indeed have that, then hash tables are probably the next important thing to implement. The increase in speed you get is immense compared to all other improvements.

After that, opening books are very helpful, not only to prevent blunders in the opening but also to prevent using valuable time on the clock while your opponent makes 10-30 moves without using any time at all.

Null move is also very helpful for a nice boost in speed.

Not knowing what you've already implemented, those are my next suggestions.

jm

[hgm]

A recent thread in the tournament forums has some items that I don't understand like:

128mb hash each
3-4-5 piece tablebases
Charlotte 1.0 mini.cgb book

128mb hash each is the sizeof the hash table that theenginewill be allowed to use. If your engine responds to the UCI "Hash" option or the WinBoard "memory" command, this should be automatic.

The tablebases indicate which tablebases will be present on the tournament machine for the engine to probe. Itdoes not specify if theyare in Nalimov or Gaviota format, so the spec is not complete. Because tablebases have no effect on engine strength, you don't have to worry much if your engine does not use tablebases at all, and in that case this information is also not relevant foryout

The opening book specified wil be handled by the GUI, so your engine will never know it. Important is that it will in general be able to have moves fed to it. In UCI this isimplicit in the protocol, for WinBoard it is essential your engine implements the "force" command.

[stevemulligan]

When it says "128mb hash each" does that mean just the TT's max size, or is that total memory footprint of the engine? I would need to implement the xboard memory command I guess. How is this enforced though if it's not the total memory footprint?

What about opening books? I don't see a command for opening books in the xboard docs. I assume the GUI will choose commands from it's book until it runs out. Can I typically still use my own built-in opening book?

What would be a time control violation be for 10 moves in 10 minutes? If I take 1 minute 5 seconds to move, does that mean I'm out? Or do I have the full 10 minutes to "catch up" before I'm out?


The engines that I play against to get my base ELO - Do I need to know their elo's before I start? Do I need to play a weaker engine during this process to get a useful rating (how applicable could it be if I loose every game?)

[hgm]

Most operating systems do not support any way to enforce a limitation on memory use. The XBoard memory command is supposed to include the size of all tables (main hash, pawn hash, tablebase cache, bitbase storage). No one will probably kill you if you use 1MB more than the maximum.

If engines have private books is their own business in WB protocol. Some rating testers don't allow you to use private books. If you want a private book that the user can configure, you could use an engine-defined option for that (e.g. feature option="Book File -file mybook.bin").

10 moves in 10 min is a 'classical' TC, and means that it is checked every 10 min if you have done enough moves, but otherwise are free to divide the ime as you want. E.g. you could think 9 min on the first move, and then to 9 in 5 sec each. Or you could start wit 19 moves in the first sec (e.g. from book), and then thing 19:50 over the 20th move.

For ratings: yes, you must know the ratings of the opponents, and should use a group of opponents against which you score between 20 and 80%. Like you say, if you always lose, it hardly tells you anything.

[UncombedCoconut]

Because tablebases have no effect on engine strength, you don't have to worry much if your engine does not use tablebases at all, and in that case this information is also not relevant for you.

Mostly true! But it seems to me that in live tournament situations they really can matter. Engines are then allowed to insist on playing until mate (in contrast to the most common testing situations, where the GUI adjudicates games if both engines agree that one of them is utterly screwed), and some of the strongest engines in the world have suffered bugs/regressions that killed their ability to mate in easy endgames. I think I even remember you telling such a horror story about one of your own programs' opponents?

Of course, fixing such bugs whenever they exist is a better solution than hiding them with TBs. But why not do both now that Miguel has made it trivial to add this feature to at least the root search?

As for opening books, Steve: no one can stop you from hard-coding such information into your program. However, it might annoy testers. They'll generally force the programs they're testing to play from each position in an opening gauntlet (either ordered or picked in some fair random way from an opening set that's larger than the desired # of games per match). From their point of view, it's more fair if both programs are required to play using only their algorithms from each position. This isn't an absolute rule, though: Glass, for instance, has a hard-coded book as a fallback for when it can't find an external book file, and AFAIK nobody minds -- the book is shallower than a tester's should be (and Glass is an awesome engine).

Sorry if I've flogged a point that everyone already understood... it's a late weekend night and I'm tired.

[jdart]

The engines that I play against to get my base ELO - Do I need to know their elo's before I start? Do I need to play a weaker engine during this process to get a useful rating (how applicable could it be if I loose every game?)

All computer chess ratings are relative, not absolute. All playing against another engine does is tell you if you are relatively stronger or weaker. So you can calculate an ELO number if you want but it's only relative. If you play on the chess servers they will calculate a rating for you, but that is only valid within their own rating pool.

--Jon

[stevemulligan]

So for time control I'll need "st" and "level" - What about time and otim. Do I need those too?

What about insufficient material? What is the safest way to claim a draw that works with most GUI's? If I claim a draw and the GUI doesn't think there is insufficient material my guess is that would count as a loss?