opening book standards

[Dann Corbit]

Polyglot is a bad book format.
It does not carry enough information.
The interface to find the position is good.
But what was the depth of the analysis for this position?
What was the number of wins, losses, and draws in history?
What was the number of wins, losses, and draws for my program?

It's way too naked to be a good book format. It's nothing but the move to make.

Of course, most books are badly broken like that.

[Dann Corbit]

Polyglot books are actually a pretty old standard. It was never changed, at least as far as Chess is concerned. (Although the standard was extended in a backward-compatible way to allow header info in the book. WinBoard does not use that yet, though.)

I don't know what the '1337 Polyglot book' is, and I don't know anything about SCID. It is not clear to me what you mean by "shown in SCID". Did the book file not show up in the file-browse dialog, or was SCID unable to find any moves in it? If WinBoard could load the bookand find moves in it,it must be a regular Polyglot book, and I cannot think of any reason why it would not work in SCID.

Scid is a tcl/tk based chess interface. It was designed for correspondence chess use. You can analyze with it. It is portable to Windows and Linux.

1337 was a chess program, so I presume that the 1337 book was made for that program. But of course, you can use the books with any program

[hgm]

Polyglot is a bad book format.
It does not carry enough information.
The interface to find the position is good.
But what was the depth of the analysis for this position?
What was the number of wins, losses, and draws in history?
What was the number of wins, losses, and draws for my program?

It's way too naked to be a good book format. It's nothing but the move to make.

Of course, most books are badly broken like that.

That is not entirely true. Polyglot books contain a weight to each move, representing the probability the move should be played. In the end that is all it boils down to, when you use anopening book. No matter howmuch info is contained in the book, or how complex the algorithm that you grind it through to decide upon a move from it, it will result in a reproducible probability for that move to be played. You might as well just have stored that probability in the book to begin with. Which is exactly what Polyglot does.

This is the difference between a 'cooked book' and a 'raw book'. Having the raw info just requires the book user to do alot ofprocessing to finally obtain what a cooked book would provide in the first place.

Also note that Polyglot book format includes so-called 'learn fields' for each move, in addition to the weight. These can record how many times the book user played a move, and how many half-points he scored with it. The Polyglot adapter updates this info in the book it uses when bok learning is on, and uses the info during selection of a book move to modify the playing probabilities specified by the weights, to discourage moves with a poor performance record, and encourages playing of moves that were played less often than their weight suggests.

[Dann Corbit]

Polyglot is a bad book format.
It does not carry enough information.
The interface to find the position is good.
But what was the depth of the analysis for this position?
What was the number of wins, losses, and draws in history?
What was the number of wins, losses, and draws for my program?

It's way too naked to be a good book format. It's nothing but the move to make.

Of course, most books are badly broken like that.

That is not entirely true. Polyglot books contain a weight to each move, representing the probability the move should be played. In the end that is all it boils down to, when you use anopening book. No matter howmuch info is contained in the book, or how complex the algorithm that you grind it through to decide upon a move from it, it will result in a reproducible probability for that move to be played. You might as well just have stored that probability in the book to begin with. Which is exactly what Polyglot does.

This is the difference between a 'cooked book' and a 'raw book'. Having the raw info just requires the book user to do alot ofprocessing to finally obtain what a cooked book would provide in the first place.

Also note that Polyglot book format includes so-called 'learn fields' for each move, in addition to the weight. These can record how many times the book user played a move, and how many half-points he scored with it. The Polyglot adapter updates this info in the book it uses when bok learning is on, and uses the info during selection of a book move to modify the playing probabilities specified by the weights, to discourage moves with a poor performance record, and encourages playing of moves that were played less often than their weight suggests.

All chess books should be live books. That means, when you finish a game, you update the annotations in the book with new data. Did you win or lose? What was the score when you exited the book? You should also be able to feed the book with new data from outside sources. The idea of mini-maxing the book (as old as BOOKUP) is a very good one, which is what I think the chessbrain thing is doing.

[Dann Corbit]

Polyglot is a bad book format.
It does not carry enough information.
The interface to find the position is good.
But what was the depth of the analysis for this position?
What was the number of wins, losses, and draws in history?
What was the number of wins, losses, and draws for my program?

It's way too naked to be a good book format. It's nothing but the move to make.

Of course, most books are badly broken like that.

That is not entirely true. Polyglot books contain a weight to each move, representing the probability the move should be played. In the end that is all it boils down to, when you use anopening book. No matter howmuch info is contained in the book, or how complex the algorithm that you grind it through to decide upon a move from it, it will result in a reproducible probability for that move to be played. You might as well just have stored that probability in the book to begin with. Which is exactly what Polyglot does.

This is the difference between a 'cooked book' and a 'raw book'. Having the raw info just requires the book user to do alot ofprocessing to finally obtain what a cooked book would provide in the first place.

Also note that Polyglot book format includes so-called 'learn fields' for each move, in addition to the weight. These can record how many times the book user played a move, and how many half-points he scored with it. The Polyglot adapter updates this info in the book it uses when bok learning is on, and uses the info during selection of a book move to modify the playing probabilities specified by the weights, to discourage moves with a poor performance record, and encourages playing of moves that were played less often than their weight suggests.

All chess books should be live books. That means, when you finish a game, you update the annotations in the book with new data. Did you win or lose? What was the score when you exited the book? You should also be able to feed the book with new data from outside sources. The idea of mini-maxing the book (as old as BOOKUP) is a very good one, which is what I think the chessbrain thing is doing.

I think that the polyglot book would be much nicer if you could insert new rows on the fly (which may be possible, I did not try it). If you can put all the pv nodes in your hash table into the book after each game, then it would actually be a pretty good format. I was not aware of the learning feature. I guess I have not been keeping up very well. That is also very important.

[Ozymandias]

in Inboard i could see this free 1337 polyglot book, but not in scid

I haven't tried WB, but ScidVPC, and Scid by extension, should allow you to see/edit the moves. If you can see them choosing other books, but you can't when selecting that one from the menu... that would be a first for me.
Did you place it in the default folder for books?

[Ozymandias]

It does not carry enough information.
But what was the depth of the analysis for this position?
What was the number of wins, losses, and draws in history?
What was the number of wins, losses, and draws for my program?

For the engine to decide what move to make, it has enough, for anything else, a DB will always be better.
Analysis? That might be cerebellum.
They will be between the parameters stablished, at the time of making the book.
I don't think any book will tell you that, again, you want a DB.

[Michel]

Polyglot books are actually a pretty old standard. It was never changed, at least as far as Chess is concerned. (Although the standard was extended in a backward-compatible way to allow header info in the book. WinBoard does not use that yet, though.)

I don't know what the '1337 Polyglot book' is, and I don't know anything about SCID. It is not clear to me what you mean by "shown in SCID". Did the book file not show up in the file-browse dialog, or was SCID unable to find any moves in it? If WinBoard could load the bookand find moves in it,it must be a regular Polyglot book, and I cannot think of any reason why it would not work in SCID.

GnuCheese does some sanity checking when opening a book (it checks if the first 10 key fields are non decreasing). This works quite well. The code is actually in the file pgheader.c in pgheader_detect() in the polyglot source.

The header format I was proposing long ago also specified a magic string

\x00\x00\x00\x00\x00\x00\x00\x00@PG@\x0a

There is a corresponding magic entry so that on Linux the OS can recognize a polyglot book.

Code: Select all

#------------------------------------------------------------------------------
# polyglot:  file(1) magic polyglot chess opening book files
#
# From Michel Van den Bergh 

0       string          \x00\x00\x00\x00\x00\x00\x00\x00@PG@\x0a           Polyglot chess opening book
>13     string          1.0\x00\x00\x00\x00\x00\x00\x00\x00\x0a                (version 1.0)
!:mime  application/x-polyglot

[jefk]

[quote="Ozymandias"][quote="jerk"]
Did you place it in the default folder for books?[/quote]

sure, even renamed it, removed the spaces in the name;
this is the book i was looking at:
http://1337chess.blogspot.nl/2013/01/13 ... yglot.html
free version of a commercial book; but i don't care so much
about Scid, just looked at the version, and it was 4.4; so i will
have to upgrade; in ScidVsPC i can see the book without any problem;
even can use two books (but with polyglot you also can merge books)

Anyway about polyglot books, if it works in winboard then it's a polyglot
book says Harm, and apparently he now is happy to take 'responsibility'
for polyglot (updates; which will not be needed anyway (?))..

Then about (book)standards again, i agree with Harm that
for playing you don't need more info then what polyglot
provides; yet if you want to make a (polyglot or other)
book from various sources, it needs various conversion tricks;
such sources, eg. db's sometimes have the advantage
for opening instruction/education, by showing lines/
tables, games (Chessbase) or being able to minimax
(Cerebellum, Aquarium, Bookbuilder).

There exist(ed) various tools for converting book formats,
abk2pgn, now even with little Gui, FGCRWNEW
http://www.autochess.com/forum/convert- ... -t872.html
(for which i can't find a proper download link anymore; also
couldnt find the engine 'thinker' and its booktool 'bookthinker)

But for .ctg to pgn i now found this one:
https://github.com/sshivaji/ctgexporter
but it has to be compiled with the makefile (anyone?)

The more complicated Aquarium book formats (which can
combine .ctg, cap, and minimaxed trees) can be used
in an Uci interface in a similar way as polyglot, with
their book converter
http://www.autochess.com/forum/new-aqua ... html#p6386

Ideally, this Aquarium book converter should simply
convert to book to a Polyglot format and then we have
one standard..same for the Cerebellum tool ofcourse

jef

PS finally i should mention an academic standards for
chess opening books, google engine Dame Blanche
(and a tool for their books 'book-builder') and you'll find
more info; interesting maybe, but not better
than polyglot i think, so i'll discard it.

[Michel]

Ah.

I see that the polyglot books created by HGM's version of polyglot (the main version) now also include the header (before it was only my version that did this)! The code is in book_make.c starting at line 356.

http://hgm.nubati.net/cgi-bin/gitweb.cg ... 51787f0d8f

Thx.

The header format (which is very basic, but extensible) is discussed here (including sample code)

http://hardy.uhasselt.be/Toga/pgheader- ... eader.html

I will submit the magic string mentioned above to the file mailing list to make it official.

http://mx.gw.com/pipermail/file/

EDIT: It is really nice to be able to do this

Code: Select all

$ file gnuchess.bin 
gnuchess.bin: Polyglot chess opening book (version 1.0)