Polyglot extension tool

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
User avatar
Rebel
Posts: 4515
Joined: Thu Aug 18, 2011 10:04 am

Polyglot extension tool

Post by Rebel » Sun Jan 06, 2019 3:18 pm

Poly version 1.0

The polyglot book format is the most widespread format in use by chess engines. The format contains 4 unused bytes. Have a look at the nice things you can do using those bytes.

http://rebel13.nl/download/books.html
Everybody is unique, except me.

User avatar
hgm
Posts: 23376
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: Polyglot extension tool

Post by hgm » Sun Jan 06, 2019 7:52 pm

Beware that Polyglot uses these bytes for book learning, and that they have quite different meaning in that case. So if you are using a book of the described format, be sure book learning stays switched off, or the extra info will be corrupted.

In Polyglot books produced by WinBoard in Monte-Carlo book mode these bytes are used to store WDL counts (which are not deducible from the book weights alone), so that you can load an existing book and add continue the MC play.

User avatar
Rebel
Posts: 4515
Joined: Thu Aug 18, 2011 10:04 am

Re: Polyglot extension tool

Post by Rebel » Sun Jan 06, 2019 10:12 pm

I thought about the same but did not like the maximum of 1023 (10 bits) of the WLD counters. Or were you able to increase the maximum?
Everybody is unique, except me.

User avatar
hgm
Posts: 23376
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: Polyglot extension tool

Post by hgm » Mon Jan 07, 2019 10:11 am

It appears I actually cheat here, by only keeping W+D and L+D, each in 16 bit. I originally implemented the Monte-Carlo book building in WinBoard for making a mini-Shogi book. And draws are not possible in mini-Shogi, so I added these as an afterthought.

Note, however, that the weight field in a Polyglot book normally contains 2*W+D. So the three fields together make it possible to construct the full WDL info: W = weight - learnPoints; D = learnPoints - W; L = learnCount - D;

User avatar
Ozymandias
Posts: 1087
Joined: Sun Oct 25, 2009 12:30 am

Re: Polyglot extension tool

Post by Ozymandias » Mon Jan 07, 2019 10:41 am

Rebel wrote:
Sun Jan 06, 2019 3:18 pm
Poly version 1.0

The polyglot book format is the most widespread format in use by chess engines. The format contains 4 unused bytes. Have a look at the nice things you can do using those bytes.

http://rebel13.nl/download/books.html
Are those bytes 16bit?
What are the value boundaries for Score and Depth?
What's the role of Score and Depth, in the move decision?
How does the learn byte keep track of losses? I'm guessing you use a different method, to the way weights are computed (see above).
What use is the learn value if several moves exceed 125?

User avatar
hgm
Posts: 23376
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: Polyglot extension tool

Post by hgm » Mon Jan 07, 2019 12:18 pm

Bytes are 8 bit, so 4 bytes is 32 bit. In the Polyglot book code they are used to store two 16-bit counters, learnCount and learnPoints. (Presumably the latter is actually half-points). Weight and move are also 16-bit quantities; together with 8 bytes for the position key this makes a 16-byte entry. Each move has its own entry, even if it is from the same position.

User avatar
Ozymandias
Posts: 1087
Joined: Sun Oct 25, 2009 12:30 am

Re: Polyglot extension tool

Post by Ozymandias » Mon Jan 07, 2019 1:45 pm

hgm wrote:
Mon Jan 07, 2019 12:18 pm
Bytes are 8 bit, so 4 bytes is 32 bit. In the Polyglot book code they are used to store two 16-bit counters, learnCount and learnPoints. (Presumably the latter is actually half-points). Weight and move are also 16-bit quantities; together with 8 bytes for the position key this makes a 16-byte entry. Each move has its own entry, even if it is from the same position.
Thanks, where do the "unused" bytes come from? From learnCount or learnPoints?

User avatar
hgm
Posts: 23376
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: Polyglot extension tool

Post by hgm » Mon Jan 07, 2019 3:51 pm

Yes. So 'unused' is a relative concept.

Jonathan003
Posts: 46
Joined: Fri Jul 06, 2018 2:23 pm
Full name: Jonathan Cremers

Re: Polyglot extension tool

Post by Jonathan003 » Mon Jan 07, 2019 7:47 pm

Hi I use Lucas Chess to build opening repertoires for over the board games.

I was thinking if I use a bin book as base for building a repertoire. It would be interesting to change the weight of the book moves according engine analysis.

So If I import only best move or best moves from a bin book, the moves will probably be good moves.

I would do it this way: change the results of the games to drawn games before building the bin books. (I use games from Grandmasters, engine games are to artificial for a repertoire for over the board games I think). Than in Lucas Chess choose to import all best moves for one collar from these bins. (So only the moves that where most played are imported, regardless of the results of the games)

Then export to pgn, Build new bin books from these pgn's and analyze these bin book with stockfish 10.

I the weight of the books where updated according the evaluation of the chess engine. I would maken new opening lines in Lucas Chess and choose to only import the best moves from the bin book.

I was searching on the web for a tool to just do that.

I came across poly1.0

http://rebel13.nl/download/books.html

I tried it but I don't see any changes in the weight of the moves before and after if I open the bin books in SCID

Do I do something wrong or isn't that the purpose of poly1.0
If not I would like it if someone could write a program that can do just that.

User avatar
Rebel
Posts: 4515
Joined: Thu Aug 18, 2011 10:04 am

Re: Polyglot extension tool

Post by Rebel » Tue Jan 08, 2019 12:18 am

Answered that on Facebook

Hi there,

The util doesn't change the weight, or anything, it just adds extra information to the book. At the end of the page it states programmers who support the Polyglot format need to update their code to make use of the new features.

But your comment about weights made me think. I will make 2 extra functions that will set the weight on 100% of the best Stockfish move or the highest learn value so that people are no longer dependent on programmers.

Just keep an eye on the website, an update will take not so long, one week max.
Everybody is unique, except me.

Post Reply