Creating Books from .PGN files

[Michel]

I can't see the problem with book size at first read through book_make.cpp

I have forgotten what the exact problem is but it has to do with a 32 bit index
that overflows somewhere.

[Dave_N]

Oh yes I remember I have seen the code, I thought of a
"Binary Search Tree " instead and did not understand where it would be implemented ... some kind of Hash collision in my own memory I think

[Dave_N]

I can't see the problem with book size at first read through book_make.cpp

I have forgotten what the exact problem is but it has to do with a 32 bit index
that overflows somewhere.

I have DDD for debugging with Cygwin. I suppose I would need to attempt to build a large book first ... how many games are needed before Polyglot will crash?

[yoshiharu]

To assign a score to a move in the book I use the expected score ( i.e. #wins+#draws/2 divided by #games) multiplied by the average number of "non-losses" (i.e. #wins+#draws divided by #games); the idea is that if a move leads to fewer draws is less reliable, possibly some refutation of the move is lying around, or maybe the position is very doubly-edged.
I must admit it's a conservative view.

What I am really after, though, is a sort of minimax on the opening book, because I am sick and tired of seeing my engine entering some continuation whose stats look great before a critical move, but after that the opponent has a move that *inverts* the stats refuting the whole plan...
Only problem is that I don't know what evaluation to use in this restricted minimax, using the usual stats seems too naive...

Cheers, Mauro

[Michel]

I think the problem was with the mallocs in book clear.

However it might have been with a 32 bit system. The argument to
malloc is size_t. I assume this is 64 bit on a 64 bit system. So then
there should be no problem.

Code: Select all

static void book_clear() {

   int index;

   Book->alloc = 1;
   Book->mask = (Book->alloc * 2) - 1;

   Book->entry = (entry_t *) my_malloc(Book->alloc*sizeof(entry_t));
   Book->size = 0;

   Book->hash = (sint32 *) my_malloc((Book->alloc*2)*sizeof(sint32));
   for &#40;index = 0; index < Book->alloc*2; index++) &#123;
      Book->hash&#91;index&#93; = NIL;
   &#125;
&#125;

[Don]

I suggest that you do whatever makes sense to you here, but also make a conversion utility that will convert to polyglot format. You can ensure that the move percentages make sense since you have really good control over that.

I had a look at the Polyglot format and found that it doesn't give a win/loss/draw percentage and instead gives a weight, so I decided to have another look at creating books. My first thought was to modify the polyglot book creation to store white wins in the "weight" variable and draws or black wins in the "learn" variable. I decided to ignore polyglot and look at making my own format from my own pgn tree code...

At the moment I have written a code that will merge games into 1 game with variations, however I cannot detect transpositions. So for example if 2 games take a Ruy Lopez line with the Marshall attack and one line has black playing b5 after a6 and the other has Nf6 first (as in the mainline) then I don't have the games rejoining after the positions become equal, and instead the merged game is a variation of the first.

Obviously I could not compare FEN in a realistic time to detect the transposition so I would need some kind of hash table (probably) unless I create a FEN tree ... I am sure this has been tried before but is it practical?

[Dave_N]

I think the problem was with the mallocs in book clear.

However it might have been with a 32 bit system. The argument to
malloc is size_t. I assume this is 64 bit on a 64 bit system. So then
there should be no problem.

Code: Select all

static void book_clear() {

   int index;

   Book->alloc = 1;
   Book->mask = (Book->alloc * 2) - 1;

   Book->entry = (entry_t *) my_malloc(Book->alloc*sizeof(entry_t));
   Book->size = 0;

   Book->hash = (sint32 *) my_malloc((Book->alloc*2)*sizeof(sint32));
   for &#40;index = 0; index < Book->alloc*2; index++) &#123;
      Book->hash&#91;index&#93; = NIL;
   &#125;
&#125;

I am not sure if Cygwin will convert size_t to a 64 bit unsigned int automatically, perhaps some configuring is needed ....

[Dave_N]

Yes a conversion tool would be correct for compatibility, I will probably make a dialog based program for tweaking the weights with the default setting being the normal polyglot version.

[hgm]

What I am really after, though, is a sort of minimax on the opening book, because I am sick and tired of seeing my engine entering some continuation whose stats look great before a critical move, but after that the opponent has a move that *inverts* the stats refuting the whole plan...
Only problem is that I don't know what evaluation to use in this restricted minimax, using the usual stats seems too naive...

This is one of the things that is still on my (long) to-do list: put a good book-building algorithm into Polyglot. I think I designed a good algorithm for this (indeed using minimax-like back-propagation of the move stats). Pure minimax would only be valid if you prepare a book for 'best-move-only' play. If you want to switch on variety, you will also have to play non-best moves with a finite probability. E.g. you could reduce the probability of the move to be played by a factor 0.9 for every percent it scores less. But that means that the score of a position that has a move that scores 53% and two moves that score 50% is not is not 53% (as it would be in minimax) but (1*53% + 0.729*50% + 0.729*50%)/(1+0.729+0.729). This is worse then the stats of the best move, because the others drag it down, but better than the raw stats of the position, because the inferior moves have lower weight.

[Dave_N]

shouldn't there be a probability 0 for an engine to play a move that scores 75% wins for the opponent?