Opening books?

[mike_bike_kite]

I was under the mistaken impression that not having an opening book would lead to more varied play by my little program. Sadly the opposite seems to be true. Are there any open source and preferably very small opening books available? What format would they come in?

[tpetzke]

Hi, the algorithm used in chess programming are deterministic, so if you search for a best move equal input will result in equal output.

An opening book will not help you because if you have a deterministic algorithm that chooses a move always the same move will be chosen unless you factor some randomization in.

But you can do this also without a book, just modify the thinking time for your applet by a small random amount of time. Even if the first move played is still the same, the hash table will contain some different entries so with every move chances increase that the engine leaves the previous path of play.

The only opening book standard I know is the one from polyglot

http://alpha.uhasselt.be/Research/Algeb ... ormat.html

but I can't comment on it as I haven't implemented a book yet, and if I do I will probably build my own private one. I don't like the idea to pollute the engine with code and data produced by others. This applies to table bases and opening books just as it applies to copied code.

Thomas...

[Michel]

Just implement Polyglot opening books. See here for the format and
some freely available example code

http://alpha.uhasselt.be/Research/Algeb ... ormat.html

(or look at the Fruit or Stockfish source code).

Then you immediately have hundred's of books to choose from.

[mike_bike_kite]

Thanks for the suggestions guys. Must admit I'm quite confused by the approach taken though and hope you might be able to enlighten me. I imagined the opening book would just be a list of moves with an associated name. The program would then just match current moves (if any) to the available openings and, if it has any matches, pick one and just play the next move. So if the program opened d4 and the opponent replied Nf6 then it could pick from any of the following continuations:

Code: Select all

	static String Openings[] = {
		"d4 Nf6 c4 e6 g3: Catalan Opening",
		"d4 Nf6 Nf3 e6 Bg5: Torre Attack",
		"d4 Nf6 c4 c5 d5 b5: Benko Gambit",
		"d4 Nf6 c4 c5 d5 e6: Benoni Defense",

The code supplied seems to store the openings in an unreadable (by human eyes) and I just can't see why. There's no speed advantage and if one opening proves unsuitable for your program then you'd have a devil of a job to find and delete the opening. You also don't get a nice opening name to show the user.

So why is it done in this fashion?

Code: Select all

const uint64 Random64[781] = {
   U64(0x9D39247E33776D41), U64(0x2AF7398005AAA5C7), 
U64(0x44DB015024623547), U64(0x9C15F73E62A76AE2),
...

Perhaps it stops the really determined opponent from looking at the source code to find what openings it's missing?

[hgm]

By storing the moves per position you automatically catch transpositions, because the position would be recognized irrespective of the path it was reached through. It is also more compact to encode a position by a hash key (even if that is 8 bytes, which seems like generously overdoing it) than by a sequence of moves leading to it. Also picking early moves might become very cumbersome if you are storing a list of lines: you might have 100,000 lines starting with 1. e4 c5, distributed over three follow-up moves. It is not at all said that the third move that occurs most is also the one that you would want to play most. Having complete lines makes it difficult to put openings in your book that you want one side to play, but the other to avoid. (E.g. you might not want your program to ever play 1. f4, but you would like it to know that it should play 2... e5! when the opponent happens to play it.

[lucasart]

I was under the mistaken impression that not having an opening book would lead to more varied play by my little program. Sadly the opposite seems to be true. Are there any open source and preferably very small opening books available? What format would they come in?

you needn't bother with books. that is the job of the interface. you don't need to code it, unless you really want to.
the polyglot format is the most common open source one.

[UncombedCoconut]

With polyglot opening books, you use the current position to look up candidate moves. That way, you don't have to store separate lines if there's a transposition (e.g. 1. c4 Nf6 2. Nf3 * vs 1. Nf3 Nf6 2. c4 *).

The Random64 table is used to compute a hash code from a position in a standard way. This code is the key for looking up book entries. These entries consist of a move and a "weight" (proportional to how often you should play it).

Once you understand the format, it's very easy to implement. I think I did it in a day in PyChess; it should be easier in a C/C++ engine since you can re-use more code directly.

[mike_bike_kite]

I understand and that does make sense now.

I can't resist the simplicity of having a list of moves in a string though and I like the idea of being able to name an opening - I can learn the openings as I code! At least now I know why I'm doing it wrong.

[bob]

Thanks for the suggestions guys. Must admit I'm quite confused by the approach taken though and hope you might be able to enlighten me. I imagined the opening book would just be a list of moves with an associated name. The program would then just match current moves (if any) to the available openings and, if it has any matches, pick one and just play the next move. So if the program opened d4 and the opponent replied Nf6 then it could pick from any of the following continuations:

Code: Select all

	static String Openings[] = {
		"d4 Nf6 c4 e6 g3: Catalan Opening",
		"d4 Nf6 Nf3 e6 Bg5: Torre Attack",
		"d4 Nf6 c4 c5 d5 b5: Benko Gambit",
		"d4 Nf6 c4 c5 d5 e6: Benoni Defense",

The code supplied seems to store the openings in an unreadable (by human eyes) and I just can't see why. There's no speed advantage and if one opening proves unsuitable for your program then you'd have a devil of a job to find and delete the opening. You also don't get a nice opening name to show the user.

So why is it done in this fashion?

Code: Select all

const uint64 Random64[781] = {
   U64(0x9D39247E33776D41), U64(0x2AF7398005AAA5C7), 
U64(0x44DB015024623547), U64(0x9C15F73E62A76AE2),
...

Perhaps it stops the really determined opponent from looking at the source code to find what openings it's missing?

Many of us did that (move list approach) many years ago. The problem is transpositions. Unless you include the transpositions, you will miss them and play into openings you don't like.

I remember playing against Sargon once, as they always used the move list approach after we had converted to storing the hash signature, which is actually simpler. I think the opening went something like (we were white) 1. e4 e5 2. d4 exd4 3. Nf3 where we are now back into a normal Sicilian if black plays something like Nc6. But they tried to hang on to that d-pawn for dear life, and the game was over in about 20 moves total.

Another issue is that of intentionally changing the order of the moves when it is less tricky, just to get your opponent out of book. If the moves are pretty forced for the next 20 moves, if you get your opponent out, he will burn most of his clock calculating replies to your instant book moves, and you end up with way more time than him, which is just like having a way faster program than his...

A final advantage to positions is that you can leave book on move 7 when your opponent plays something different or moves in a different order, but you can get back into the book at move 10-11-12 as you get back to known book territory via an unexpected side-trip through non-book moves. I have seen a few games where we were in and out of book 3-4 times before finally getting out for good...



I went to positions because I wanted to (a) handle transpositions and (b) not have to create long linked lists of moves that are either hard to edit, or hard to add to the book. Positions are easy to add if you want, although today's hardware makes rebuilding even a large book very fast...

[UncombedCoconut]

I can't resist the simplicity of having a list of moves in a string though and I like the idea of being able to name an opening - I can learn the openings as I code! At least now I know why I'm doing it wrong.

If you really want that, download SCID and look at the scid.eco file. The drawbacks should be obvious. Normally such files are used not as books, but as a supplement the GUI can use to give opening names. Another idea is to use cutechess-cli's -pgnin option (you won't even have to modify the engine at all), or use a PGN file to generate a book. (With the latter approach you should be careful that you get reasonable frequencies for the common and off-beat openings.)