Header format for polyglot books v2

[Michel]

When thinking about the header format proposed here

http://hardy.uhasselt.be/Toga/header.html

I noticed that it has a major flaw with respect to extensibility. So I am proposing a backward incompatible change with respect to the above specification. The change makes the header also more elegant. Please give comments.

Currently the (logical) header consists of fields, some of which are predefined and the remaining ones are free format (they should be regarded as comments). The number of predefined fields is a function of the version number (which is embedded in the header).

Here is the flaw: when the version is upgraded, applications which are not aware of the new version will not know about the added fields and dump them in the free format section.

To give a concrete example. The following header represents a book for normal and suicide chess.

"@PG@\n1.0\n2\nnormal\nsuicide\n(normally comments here)"

Assume v1.1 adds a predefined field xxx. Then the header becomes

"@PG@\n1.1\n2\nnormal\nsuicide\nxxx\n(normally comments here)"

A v1.0 application will interpret xxx as a comment. This may seem harmless but users will not know what xxx means and perhaps delete it. With potentially bad consequences for a v1.1 application which expects the field to be there.

So my suggestion is to put the number of predefined field in the header itself (other than the magic and the version number).

Thus the v1.0 version of the above header will become

"@PG@\n1.0\n3\n2\nnormal\nsuicide\n(normally comments here)"

And the v1.1 version

"@PG@\n1.0\n4\n2\nnormal\nsuicide\nxxx\n(normally comments here)"

A v1.0 application will now simply skip the field xxx which it does not know about.

Here is the Backus Naur form of the proposed changed header. As you can see
logically it is now a tree.

Code: Select all

<header>        &#58;= <magic>\n<version>\n<root-field>&#91;\n<field>&#93;*
<magic>         &#58;= @PG@
<version>       &#58;= <number>.<number>
<root-field>    &#58;= <number>&#91;\n<multi-field>&#93;*
<multi-field>   &#58;= &#91;<root-field> | <field>&#93;
<field>         &#58;= <string>

[Michel]

Sorry the v1.1 version would of course be

"@PG@\n1.1\n4\n2\nnormal\nsuicide\nxxx\n(normally comments here)"

[Michel]

The beauty of nicely refactored software! It took me only 15 minutes to implement the proposed changed header without needing to change any api's.

http://hardy.uhasselt.be/Toga/pgheader_v2.tar.gz

[Michel]

Here is an update version of the header proposal:

http://hardy.uhasselt.be/Toga/header_v2.html

[Daniel Shawul]

I just want to give a thumbs up for your work. Don't have much to comment, but I like the free format section where you can put comments ,copyright info etc. Still have doubts with multi-variant book : difficult post-processing, probably a confusing comment section if different authors are involved in creating the books, and a difficult format (already have additional flag indicating number of variants). IMO if you can do it separate, it is better not to mix to keep the format as simple as possible. Anyway the format allows for creating of a single book for a single variant, so authors don't have to mix variants when creating books. But I am afraid GUI authors will not be motivated to support multi book loading when an easy option of loading a single book with all variants is provided. So in effect only multi books will be supported..
Edit An idea that reinforces my belief: If it was endgame tablebases for different variants we are talking about, then it becomes clear mixing variants in one file would be a very bad. So I think on recieving a new variant through 'variant YYY' the engine/gui should reset everything including book or egtb loading.

[Michel]

But I am afraid GUI authors will not be motivated to support multi book loading when an easy option of loading a single book with all variants is provided. So in effect only multi books will be supported..

The interface I had in mind for icsdrone was that the user would provide a list of books (through the -book option). icsdrone would pick the first book in the list that supports the current variant. In that way both muli-books and single books would be supported....

[Michel]

I have now released version 1.0 of the specification as well as version 1.0
of my utility. Both can be found here

http://hardy.uhasselt.be/Toga/pgheader-release/

(there is a windows binary).

I have also upgraded the last version of Polyglot I have been maintaining (1.4.67b) to one that is aware of the new header (1.4.70b).

polyglot make-book

will write a default header and

polyglot merge-book

will merge the variant lists in the books it is merging.

Version 1.4.70b can be found here:

http://hardy.uhasselt.be/Toga/polyglot-release/

[ZirconiumX]

I suggest we have a new extension for the new polyglot books so that people don't put a headered book in a chess engine that doesn't support it.

.bin2 and .bv2 are the ones that come up to my head first.

Matthew:out

[Michel]

I suggest we have a new extension for the new polyglot books so that people don't put a headered book in a chess engine that doesn't support it.

A chess engine does not have to support the header... The header is invisible unless the chess engine makes a special effort to read it.

I don't like the .bin extension either as it is a highly overloaded extension. I had to jump through hoops to make my webserver serve gnuchess.bin with the mime type "application/x-polyglot" (by default .bin is "application/octet-stream"). But even now that it is served correctly Firefox still insists it is simply a "binary file". Only konqueror pays attention to the mime type.

This being said I think it would be hard to get an agreement on a new extension as too few people care. And anyway this is orthogonal to the header specification.