Header format for polyglot books

[Michel]

I am starting a new topic since the issue of polyglot metadata is also important for people not interested in variants.

Historically polyglot books have contained no metadata. This is problematic for many reasons. For example the file type cannot be reliably identified by the OS and it is impossible to embed author/copyright/license information.

The following document proposes the addition of an extensible fully backwards compatible header to polyglot book files.

http://hardy.uhasselt.be/Toga/header.html

To facilitate implementation, free to use, sample code is provided to add, display and delete such headers.

Please feel free to comment.

[mcostalba]

Please feel free to comment.

How can you be sure that an opening position doesn't have zero key ? IOW what about hash collisions ?

P.S: In case someone plans to answer something along the lines of "the probabilty of this is very very small", "doens't happens in practice", etc please do me a favor and be so kind to post instead "collision avoidance is not guaranteed". Thanks in advance.

P.P.S: Giving the above, perhaps the "move" field should keep a not valid move value, for instance where from square equals destination square.

[Michel]

How can you be sure that an opening position doesn't have zero key ? IOW what about hash collisions ?

One cannot be sure but the probability is infinitesimally small. Of course polyglot and other bookmaking programs should be thought not to write
records with keys 0x0 (so as not to overwrite the header) but this is only a theoretical issue.

[ZirconiumX]

IIRC, the 0x0 magic maps to
[d]8/8/8/8/8/8/8/8 w - - 0 1
which would never happen in a FIDE-compliant game of chess but I have no idea about other variants.

Matthew:out

[Michel]

IIRC, the 0x0 magic maps to
[d]8/8/8/8/8/8/8/8 w - - 0 1
which would never happen in a FIDE-compliant game of chess but I have no idea about other variants.

Nice point! Luckily it will not cause a hash collision in other variants either since for non FIDE chess the keys are xor'ed with a variant key which is non-zero.

[hgm]

Collision avoidance is not guarantee in Polyglot books. This holds for collisions between true positions, so it doesn't seem worth to worry about collisions with the meta-data key. It is just 1 key out of the many that are in the book. For any collision the book move could be invalid in the position for which you probe the book.

[mcostalba]

Collision avoidance is not guarantee in Polyglot books. This holds for collisions between true positions, so it doesn't seem worth to worry about collisions with the meta-data key. It is just 1 key out of the many that are in the book. For any collision the book move could be invalid in the position for which you probe the book.

Exactly, so meta-data key. 's 'move' should be an invalid move to be sure is not erroneously tried up by engines in case of hash collision. For instance at what 'move' corresponds your @PG@ header? (note that a Polyglot book stores numbers in big-endian format)

[mcostalba]

One cannot be sure but the probability is infinitesimally small. Of course polyglot and other bookmaking programs should be thought not to write
records with keys 0x0 (so as not to overwrite the header) but this is only a theoretical issue.

Of course? If you claim your extension is 100% back compatible then polyglot and other bookmaking programs should be thought nothing.

[hgm]

Exactly, so meta-data key. 's 'move' should be an invalid move to be sure is not erroneously tried up by engines in case of hash collision. For instance at what 'move' corresponds your @PG@ header? (note that a Polyglot book stores numbers in big-endian format)

I don't see the logic of that. Collisions with real positions could also come up with a move that happens to be valid. If that happens, the move will be played even if it is the worst blunder, because there is no way for the probing code to know that a collision took place. If @PG@ happens to be a valid move in some position, it would be similarly played. So what? It might do far more damage if it was an illegal move, and old probing code (not testing for this) was used to probe the book.

[mcostalba]

I don't see the logic of that

Ok I give up with this thread, we are talking different languages. Sorry for the noise.