Polyglot - merge-book oddity

[Rebel]

I did a stress test with 2 large equal books of 100Mb and the result was a 20Kb output book

With 5000 it was okay.

[Rebel]

Those who want to try the new Polyglot Merge-Books util - http://rebel13.nl/dl/Merge-Books.7z

book_merge.cpp included.

[hgm]

Finding a trick to make the symptoms of an obviously bad problem go away does not automatically cure the problem.

I suppose an 'OK' result for merging two equal 100MB books would be a 100MB book (with just all weights doubled). That is 5000 times larger than the 20K result.

Don't you think it is suspect that by increasing the buffer size 10 times, it can retain 5000 times as many entries?

[Rebel]

What I said, it's a stress test, nobody should do that, but I am pretty sure it saves me a couple of user complaints.

[Guenther]

Well, I uploaded the already compiled fixed version from HGM here:
https://drive.google.com/drive/folders/ ... E15NjVyY2M

I will remove this upload now, as HGM has done more essential fixes later anyway.
If there is still interest I could compile the newer fixes and upload again.
Just make a request here or via PM.

[jefk]

Guenther wrote

uploaded the already compiled fixed version from HGM here:

yep, and i tried it, but -besides the 'size' /duplicates issue another
flaw has risen, when i added two small books, in the new result
the percentages (of bookmoves) didn't add up to 100 pct, in fact much
smaller (it was ok in the previous -yet not perfect- version ...)
Same problem as 'Rebel reported i guess, so the latest update
doesn't have this issue anymore apparently (haven't tried it yet)

As for compiling, my code::blocks program doesn't read makefiles, guess i
have to download mingw and the 'make' program might do that later.

[hgm]

yep, and i tried it, but -besides the 'size' /duplicates issue another
flaw has risen, when i added two small books, in the new result
the percentages (of bookmoves) didn't add up to 100 pct, in fact much
smaller (it was ok in the previous -yet not perfect- version ...)

That cannot possibly be a result of any merge, as there are no percentages stored in bin books at all. The moves just have a weight (which during creation from PGN is set to the number of half-points scored with the move), and the weights of all moves from a given position can sum to anything. It is the task of the probing code to convert the weights as a percentage of the total weight.

[Joost Buijs]

If I recall the weight-field is just 16 bit (unsigned), if you start merging books without taking this into account things will probably go wrong.

Most people want to throw tens of million games at it, so you always have to re-scale the weights to 16 bit.

[Rebel]

If I recall the weight-field is just 16 bit (unsigned), if you start merging books without taking this into account things will probably go wrong.

Previously, before the new version in this thread the weights in book-1 remained unchanged. With the new version they change. It's not such a big problem because:

With the POLY util you have several options to recalculate the weights. Secondly (and not everybody will agree) the weight is pretty much out-dated, relying on engine score is superior to the weight. Last, other testing has showed a Polyglot book based on WDL is also superior to the current Polyglot weight system of 2*win+draws, ignoring the losses. All in all, Polyglot by Fabien with all its features was ahead of its time but in the meantime it can use a serious update regarding strength.

[hgm]

Weights purely from statistics are flawed anyway. You would have to minimax the opening tree in MCTS fashion to correctly judge the moves.