I'm wondering what features my engine needs to be tournament ready for normal chess in the average tournament.
A recent thread in the tournament forums has some items that I don't understand like:
128mb hash each
3-4-5 piece tablebases
Charlotte 1.0 mini.cgb book
Also, before I start tuning my engine I want to get a baseline ELO. What is the best way to go about that?
Tournament readiness
Moderators: hgm, Rebel, chrisw
-
- Posts: 117
- Joined: Wed Jul 20, 2011 2:54 pm
- Location: Ottawa, Canada
-
- Posts: 1342
- Joined: Wed Mar 08, 2006 9:41 pm
- Location: Morgantown, WV, USA
Re: Tournament readiness
I would just have it play a bunch of games against other engines and make a pgn database. Then feed that into bayselo program to get a elo rating.
Congrats on the engine
-Josh
Congrats on the engine
-Josh
-
- Posts: 1357
- Joined: Wed Mar 08, 2006 10:15 pm
- Location: San Francisco, California
Re: Tournament readiness
For "tournament readiness", IMHO the most important thing you need is stability under whatever communication protocol, GUI and time control the tournament director is going to be using. And the last of these is probably the most important. I've seen dozens of games lost by engines that didn't properly support the time control, and many others by engines that just crashed.stevemulligan wrote:I'm wondering what features my engine needs to be tournament ready for normal chess in the average tournament.
A recent thread in the tournament forums has some items that I don't understand like:
128mb hash each
3-4-5 piece tablebases
Charlotte 1.0 mini.cgb book
Also, before I start tuning my engine I want to get a baseline ELO. What is the best way to go about that?
Strength will eventually come when you implement hash tables, tablebases, opening books, and other typical improvements. But without a stable engine, it is impossible to truly determine how strong your engine is.
Just my two cents....
jm
-
- Posts: 1357
- Joined: Wed Mar 08, 2006 10:15 pm
- Location: San Francisco, California
Re: Tournament readiness
Of course, in my haste to give my two cents, I didn't consider the possibility that you already have a stable engine that supports all time controls. If you do indeed have that, then hash tables are probably the next important thing to implement. The increase in speed you get is immense compared to all other improvements.JVMerlino wrote:For "tournament readiness", IMHO the most important thing you need is stability under whatever communication protocol, GUI and time control the tournament director is going to be using. And the last of these is probably the most important. I've seen dozens of games lost by engines that didn't properly support the time control, and many others by engines that just crashed.stevemulligan wrote:I'm wondering what features my engine needs to be tournament ready for normal chess in the average tournament.
A recent thread in the tournament forums has some items that I don't understand like:
128mb hash each
3-4-5 piece tablebases
Charlotte 1.0 mini.cgb book
Also, before I start tuning my engine I want to get a baseline ELO. What is the best way to go about that?
Strength will eventually come when you implement hash tables, tablebases, opening books, and other typical improvements. But without a stable engine, it is impossible to truly determine how strong your engine is.
Just my two cents....
jm
After that, opening books are very helpful, not only to prevent blunders in the opening but also to prevent using valuable time on the clock while your opponent makes 10-30 moves without using any time at all.
Null move is also very helpful for a nice boost in speed.
Not knowing what you've already implemented, those are my next suggestions.
jm
-
- Posts: 27796
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: Tournament readiness
128mb hash each is the sizeof the hash table that theenginewill be allowed to use. If your engine responds to the UCI "Hash" option or the WinBoard "memory" command, this should be automatic.stevemulligan wrote:A recent thread in the tournament forums has some items that I don't understand like:
128mb hash each
3-4-5 piece tablebases
Charlotte 1.0 mini.cgb book
The tablebases indicate which tablebases will be present on the tournament machine for the engine to probe. Itdoes not specify if theyare in Nalimov or Gaviota format, so the spec is not complete. Because tablebases have no effect on engine strength, you don't have to worry much if your engine does not use tablebases at all, and in that case this information is also not relevant foryout
The opening book specified wil be handled by the GUI, so your engine will never know it. Important is that it will in general be able to have moves fed to it. In UCI this isimplicit in the protocol, for WinBoard it is essential your engine implements the "force" command.
-
- Posts: 117
- Joined: Wed Jul 20, 2011 2:54 pm
- Location: Ottawa, Canada
Re: Tournament readiness
When it says "128mb hash each" does that mean just the TT's max size, or is that total memory footprint of the engine? I would need to implement the xboard memory command I guess. How is this enforced though if it's not the total memory footprint?
What about opening books? I don't see a command for opening books in the xboard docs. I assume the GUI will choose commands from it's book until it runs out. Can I typically still use my own built-in opening book?
What would be a time control violation be for 10 moves in 10 minutes? If I take 1 minute 5 seconds to move, does that mean I'm out? Or do I have the full 10 minutes to "catch up" before I'm out?
The engines that I play against to get my base ELO - Do I need to know their elo's before I start? Do I need to play a weaker engine during this process to get a useful rating (how applicable could it be if I loose every game?)
What about opening books? I don't see a command for opening books in the xboard docs. I assume the GUI will choose commands from it's book until it runs out. Can I typically still use my own built-in opening book?
What would be a time control violation be for 10 moves in 10 minutes? If I take 1 minute 5 seconds to move, does that mean I'm out? Or do I have the full 10 minutes to "catch up" before I'm out?
The engines that I play against to get my base ELO - Do I need to know their elo's before I start? Do I need to play a weaker engine during this process to get a useful rating (how applicable could it be if I loose every game?)
-
- Posts: 27796
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: Tournament readiness
Most operating systems do not support any way to enforce a limitation on memory use. The XBoard memory command is supposed to include the size of all tables (main hash, pawn hash, tablebase cache, bitbase storage). No one will probably kill you if you use 1MB more than the maximum.
If engines have private books is their own business in WB protocol. Some rating testers don't allow you to use private books. If you want a private book that the user can configure, you could use an engine-defined option for that (e.g. feature option="Book File -file mybook.bin").
10 moves in 10 min is a 'classical' TC, and means that it is checked every 10 min if you have done enough moves, but otherwise are free to divide the ime as you want. E.g. you could think 9 min on the first move, and then to 9 in 5 sec each. Or you could start wit 19 moves in the first sec (e.g. from book), and then thing 19:50 over the 20th move.
For ratings: yes, you must know the ratings of the opponents, and should use a group of opponents against which you score between 20 and 80%. Like you say, if you always lose, it hardly tells you anything.
If engines have private books is their own business in WB protocol. Some rating testers don't allow you to use private books. If you want a private book that the user can configure, you could use an engine-defined option for that (e.g. feature option="Book File -file mybook.bin").
10 moves in 10 min is a 'classical' TC, and means that it is checked every 10 min if you have done enough moves, but otherwise are free to divide the ime as you want. E.g. you could think 9 min on the first move, and then to 9 in 5 sec each. Or you could start wit 19 moves in the first sec (e.g. from book), and then thing 19:50 over the 20th move.
For ratings: yes, you must know the ratings of the opponents, and should use a group of opponents against which you score between 20 and 80%. Like you say, if you always lose, it hardly tells you anything.
-
- Posts: 319
- Joined: Fri Dec 18, 2009 11:40 am
- Location: Naperville, IL
Re: Tournament readiness
Mostly true! But it seems to me that in live tournament situations they really can matter. Engines are then allowed to insist on playing until mate (in contrast to the most common testing situations, where the GUI adjudicates games if both engines agree that one of them is utterly screwed), and some of the strongest engines in the world have suffered bugs/regressions that killed their ability to mate in easy endgames. I think I even remember you telling such a horror story about one of your own programs' opponents?hgm wrote:Because tablebases have no effect on engine strength, you don't have to worry much if your engine does not use tablebases at all, and in that case this information is also not relevant for you.
Of course, fixing such bugs whenever they exist is a better solution than hiding them with TBs. But why not do both now that Miguel has made it trivial to add this feature to at least the root search?
As for opening books, Steve: no one can stop you from hard-coding such information into your program. However, it might annoy testers. They'll generally force the programs they're testing to play from each position in an opening gauntlet (either ordered or picked in some fair random way from an opening set that's larger than the desired # of games per match). From their point of view, it's more fair if both programs are required to play using only their algorithms from each position. This isn't an absolute rule, though: Glass, for instance, has a hard-coded book as a fallback for when it can't find an external book file, and AFAIK nobody minds -- the book is shallower than a tester's should be (and Glass is an awesome engine).
Sorry if I've flogged a point that everyone already understood... it's a late weekend night and I'm tired.
-
- Posts: 4366
- Joined: Fri Mar 10, 2006 5:23 am
- Location: http://www.arasanchess.org
Re: Tournament readiness
All computer chess ratings are relative, not absolute. All playing against another engine does is tell you if you are relatively stronger or weaker. So you can calculate an ELO number if you want but it's only relative. If you play on the chess servers they will calculate a rating for you, but that is only valid within their own rating pool.stevemulligan wrote: The engines that I play against to get my base ELO - Do I need to know their elo's before I start? Do I need to play a weaker engine during this process to get a useful rating (how applicable could it be if I loose every game?)
--Jon
-
- Posts: 117
- Joined: Wed Jul 20, 2011 2:54 pm
- Location: Ottawa, Canada
Re: Tournament readiness
So for time control I'll need "st" and "level" - What about time and otim. Do I need those too?
What about insufficient material? What is the safest way to claim a draw that works with most GUI's? If I claim a draw and the GUI doesn't think there is insufficient material my guess is that would count as a loss?
What about insufficient material? What is the safest way to claim a draw that works with most GUI's? If I claim a draw and the GUI doesn't think there is insufficient material my guess is that would count as a loss?