Polyglot extension tool

[Dann Corbit]

I would like to get the lookup for Epd and analysis specified as an API call of some sort.

The reason is that I have millions of position analyzed to 36 ply or better and many times more than that with shallow analysis.

So instead of looking for file 30.epd and then analyzing the raw position, why not ask for a list of already analyzed positions via API call.

e.g. expect the data to come back as {for instance}:

<Epd, char string>, <centipawn_evaluation, signed short>, <suggested_move, char[9]>, <statistical_eval, signed short>, <depth unsigned char>

Statisical eval is an estimation function based on the number of actual wins, losses, and draws for the position. There could be (for instance) a radio button for the book to average the two score numbers. Also, an extra byte for the book could be obtained by shortening suggested_move to move_number, where the possible moves are sorted lexically and the number (from 0 to 217[*]) from the position in the sorted array is the move to make. The extra space conserved could be used to store something else.

The depth field will be useful in long time control and/or superior hardware situations. If your depth is 36 and you have a 128 core machine running at 40 moves in 2 hours, you should probably discard the book value and let the computer think.

[*] The greatest mobility for any position every found is 218. But an unsigned 8 bit byte could hold up to 256 distinct values.

[hgm]

The philosophy of Polyglot book format is that it is a 'cooked' book, which directly specifies the probabilities with which the moves should be played. This as opposed to a book that contains 'raw' information (like WDL statistics), which still needs significant processing to determine what move to play.

So if there is an obvious way for determining what the effect of the engine evaluations should be on the move probabilities, it would be best to perform that calculation 'off-line', and adapt the weights accordingly. This way the book remains fully 'cooked', and no changes would be needed in the probing code of all book users.

[Rebel]

Essential update

version 1.1

Added 2 essential functions which allows the user to immediately make use of the new features without programmer support.

Weights on score - Set the weight of moves on 100% with the highest Stockfish score. The book will only play those 100% moves. Using this function basically emulates the results in table-1 above.

Weights on learn - Set the weight of moves on 100% with the highest learn value. The book will only play those 100% moves. Using this function basically emulates the results in table-2 above.

http://rebel13.nl/download/books.html

[Ozymandias]

Essential update

version 1.1

Added 2 essential functions which allows the user to immediately make use of the new features without programmer support.

Weights on score - Set the weight of moves on 100% with the highest Stockfish score. The book will only play those 100% moves. Using this function basically emulates the results in table-1 above.

Weights on learn - Set the weight of moves on 100% with the highest learn value. The book will only play those 100% moves. Using this function basically emulates the results in table-2 above.

http://rebel13.nl/download/books.html

Nice update.

As you point out in the description, one byte for the learn value is way too low to hold enough information, so that you can make a good move decision. You didn't answer, but I'm guessing that if several moves exceed the max value of 125, we have a problem. My proposal would be to use the Depth byte too, so that the max is raised to +/-32k. After all, I still don't see how the Score and Depth values could be used together in a harmonious way.

[Dann Corbit]

The philosophy of Polyglot book format is that it is a 'cooked' book, which directly specifies the probabilities with which the moves should be played. This as opposed to a book that contains 'raw' information (like WDL statistics), which still needs significant processing to determine what move to play.

So if there is an obvious way for determining what the effect of the engine evaluations should be on the move probabilities, it would be best to perform that calculation 'off-line', and adapt the weights accordingly. This way the book remains fully 'cooked', and no changes would be needed in the probing code of all book users.

The difficulty with a cooked book is that you have to completely rebuild it, you cannot simply update it with better information.
I don't think the math of wins/losses/draws would hurt the raw performance enough to even detect it.
And I think the information would make for better moves.
But I did not test my idea thoroughly. It is mostly a gedankenexperiment.

[hgm]

That is exactly why WinBoard uses the learn fields to complete the WDL info. Then it can update a book in MC book mode.

But when building a book from PGN this doesn't seem such a big deal. Even when you would just add to the PGN and rebuild from scratch it wouldn't take prohibitively long, considering how often you would do that. And there is no need to do it from scratch: you can simply build a book from the games you want to add, and use the Polyglot book merge functionality.

IMO the weakest point of the Polyglot book building process is that it equates the weights to 2*W+D, without paying any attention to losses. That means a move that was playes 10 times, with 5 wins and 5 losses, would have a smaller weight than a move that is played 100 times, with 10 wins and 90 losses. That seems wrong. Of course moves that would lose 90% of the time would not be played often in reality, if your game selection is not total crap. But still, this weighting method would underestimate promising novelties.

If the number of losses was important for the weight calculation, it would indeed not be easy to update it, unless that info was stored too.

[Rebel]

I would like to get the lookup for Epd and analysis specified as an API call of some sort.

The reason is that I have millions of position analyzed to 36 ply or better and many times more than that with shallow analysis.

So instead of looking for file 30.epd and then analyzing the raw position, why not ask for a list of already analyzed positions via API call.

e.g. expect the data to come back as {for instance}:

<Epd, char string>, <centipawn_evaluation, signed short>, <suggested_move, char[9]>, <statistical_eval, signed short>, <depth unsigned char>

I would love to make a Polyglot book from your EPD collection and then merge it with existing opening books for fexibility. In the upcoming ProDeo there will be support of the extended format and the book section will look like:

Code: Select all

From the start position:

Opening Book : books\ProDeo.bin
Positions    : 267.342

Book  Weight  Score Depth Learn
d2d4  51.60%   0.06   20   43
e2e4  41.96%   0.19   20   55
c2c4   4.13%   0.05   20   12
g1f3   2.31%  -0.01   20    0

So it can support the items I colored red.

[Rebel]

Essential update

version 1.1

Added 2 essential functions which allows the user to immediately make use of the new features without programmer support.

Weights on score - Set the weight of moves on 100% with the highest Stockfish score. The book will only play those 100% moves. Using this function basically emulates the results in table-1 above.

Weights on learn - Set the weight of moves on 100% with the highest learn value. The book will only play those 100% moves. Using this function basically emulates the results in table-2 above.

http://rebel13.nl/download/books.html

Nice update.

As you point out in the description, one byte for the learn value is way too low to hold enough information, so that you can make a good move decision. You didn't answer, but I'm guessing that if several moves exceed the max value of 125, we have a problem.

Not really because the counter can't go higher than 125 or lower than -125.

My proposal would be to use the Depth byte too, so that the max is raised to +/-32k. After all, I still don't see how the Score and Depth values could be used together in a harmonious way.

That's an interesting suggestion.

[Ozymandias]

Not really because the counter can't go higher than 125 or lower than -125.

If a position has, let's say, two move options recorded, and both saturate the learn counter (that's what I mean when I talk about exceeding 125), how do you choose a move? I see a problem there.

[Jonathan003]

Hi thanks Ed for the quick update from poly 1.0 to poly 1.1

I want to analyse a bin book with 5000 lines and 28 ply deep.

This is my hardware

Windows-versie
Windows 10 Home
© 2018 Microsoft Corporation. Alle rechten voorbehouden.
Systeem
Processor: Intel(R) Core(TM) i7-4770T CPU © 2.50GHz 2.50 G Hz
Installed memory 16,0 GB (15,9 GB available)
(RAM):
Type systeem:
64-bits besturingssysteem, x64-processor


What settings do you recommend me to use to create high valuable engine analysis of the positions? How many cores? (I think I have 4 available). What Hash size? And what time settings?

And if the proces takes to long can I interrupt the program, close my pc and proceed the analysis the next day. In other words can I do it in different steps?

If that's not possible I would like to request that for future updates. Especially because my hardware is only moderate.