Polyglot FRC/960 Opening Book

[D Sceviour]

There is not much posted on Fischer Random Castling (FRC) books, therefore it seems appropriate to start a topic for discussion. So far, there does not seem to be an FRC polyglot book.

The polyglot source files account for FRC game play, but not for FRC book making. Modifications to the original source are very easy to start compiling an FRC book. Only the book_make.c file needs to be modified. There are three changes to book_make.c:

(1) at the end of the includes add:

Code: Select all

#include "option.h"
bool board_from_fen(board_t *, const char *);

(2) add to the argument parser in book_make():

Code: Select all

      } else if (my_string_equal(argv[i],"-Chess960")) {
         option_set("Chess960","true");

(3) alter the board_start() to read FRC fen in the book_insert() sub-routine:

Code: Select all

      if (option_get_bool("Chess960")) {
         board_from_fen(board,pgn->fen);
      } else {
         board_start(board);
      }

Bingo!

It is a wonder how FRC modifications have not been made since 2014. Perhaps there is something missing or yet to be included for FRC, but there are other programmers who make changes to polyglot. If there is something else required then please post your suggestions. I do not have permission from the original polyglot author to modify and distribute copies of polyglot. Let me know if this is possible and then my altered source is available on request. For now, the executable should be renamed polyglotFRC to distinguish it from the original. In particular, Ubuntu has a download package named "sudo apt-get install polyglot" whose name interferes with compilation. Undoubtedly the package access is highly restricted so the sudo application is not recommended.

Both engine and GUI programmers will now want a starting point for FRC modifications. This example game from CCRL lists the expected FRC polyglot keys when compiling an FRC polyglot game:

[Event "CCRL 404FRC"]
[Site "CCRL"]
[Date "2012.01.06"]
[Round "98.1.101"]
[White "Stockfish 2.2.1 64-bit"]
[Black "Rybka 4.1 64-bit"]
[Result "1-0"]
[Opening "QBRKBRNN"]
[PlyCount "109"]
[SetUp "1"]
[FEN "qbrkbrnn/pppppppp/8/8/8/8/PPPPPPPP/QBRKBRNN w FCfc - 0 1"]
[WhiteElo "3192"]
[BlackElo "3161"]

1.Nf3 Ng6 2.d4 Nf6 3.c4 d5 4.cxd5 Nxd5 5.a4 Bd7 6.Bd2 O-O 7.Ng3 c6 8.O-O Rcd8
9.Qa3 a5 10.e4 Ndf4 11.e5 Bg4 12.Bxf4 Nxf4 13.Qxe7 Bxf3 14.gxf3 Ng6 15.Bxg6
hxg6 16.Qh4 Ba7 17.Rcd1 Rfe8 18.Rfe1 Qc8 19.Ne4 Qf5 20.Nd6 Qxf3 21.Nxe8 Rxe8
22.Qg3 Qf5 23.d5 cxd5 24.Rxd5 Bb6 25.Rb5 Re6 26.b3 g5 27.Rc1 Bd4 28.Rxa5 Rxe5
29.Rxe5 Bxe5 30.Qg2 g6 31.Rd1 Qf4 32.h3 Kg7 33.Qg4 Qh2 34.Kf1 Bf6 35.Qf3 Qc7
36.Kg2 b6 37.Rd5 Qc6 38.Rb5 Qc7 39.b4 Qd6 40.a5 bxa5 41.bxa5 Qd4 42.Rb6 Be7
43.Rb7 Qd6 44.a6 Qxa6 45.Rxe7 Qf6 46.Rb7 Qe6 47.Qc3 Qf6 48.Qxf6 Kxf6 49.Rb5 Kg7
50.Rxg5 Kf6 51.Ra5 Ke7 52.Kf3 Kd7 53.Ra6 Ke7 54.Ke4 Kd7 55.Ke5 1-0


When the game is pasted in a file named "FRCtest.pgn", polyglotFRC will output:

Code: Select all

user@DESKTOP:~/sources/PolyglotFRC$ ./polyglotFRC make-book -pgn FRCtest.pgn -bin FRCtest.bin -max-ply 30 -Chess960 -min-game 1

PolyGlot 1.4 by Fabien Letouzey
inserting games ...
Option Chess960 = 1
key 0x4b57ffb8744080e7
 0 move Nf3
key 0x9033134e1ca6e1da
 1 move Ng6
key 0x79daa06aeb16e550
 2 move d4
key 0xbcef8fc0fadfc812
 3 move Nf6
key 0xe701bb3557258984
 4 move c4
key 0x6b222411142a2283
 5 move d5
key 0xee48060588fe6ca5
 6 move cxd5
key 0xe0629dc82a3078b8
 7 move Nxd5
key 0x7f8ce78595d565e4
 8 move a4
key 0x14459969f846b02a
 9 move Bd7
key 0x5cff1674943a48ff
 10 move Bd2
key 0x371518f61ec39f4a
 11 move O-O
key 0xe3e9e1470971bd10
 12 move Ng3
key 0x387966db79182810
 13 move c6
key 0xaa038b340ed8fbab
 14 move O-O
key 0xa0e05713a4576173
 15 move Rcd8
key 0xaf2c271a5a7974a3
 16 move Qa3
key 0xf8c487fee0c943d6
 17 move a5
key 0x7ede323f83554f46
 18 move e4
key 0xbad93f7768d5f24c
 19 move Ndf4
key 0xfc18cd0d7fd305ee
 20 move e5
key 0x9d61dbf07bcd238a
 21 move Bg4
key 0x85a5c93a49357858
 22 move Bxf4
key 0x208a1151214f546e
 23 move Nxf4
key 0x65d71275ed64a2a2
 24 move Qxe7
key 0xb75ef9d4c57cd7c9
 25 move Bxf3
key 0x62bcb03bf5d7db9
 26 move gxf3
key 0x2c51a932bd9fefa0
 27 move Ng6
key 0xd9b877eaf572a0c
 28 move Bxg6
key 0xdf244d1f70fa9525
 29 move hxg6
1 game.
30 entries.
filtering entries ...
15 entries.
sorting entries ...
saving entries ...
all done!

Fabien Letouzey has some debug methods available inside the polyglot code but they were not used. The "key" and "move" were output using an extended information modification. The polyglot keys have been cross-tested and compared against Schooner's internal polyglot key generator - and they match! The integrity of the keys should be reliable.

How good is the book? Initial results indicate it offers a substantial increase in FRC engine strength; better than the use of books in standard chess. There is a lot of stuff for testers to keep busy with.

How easy is it to use? Theoretically, the FRC book is fully compatible with standard polyglot books. Merging with standard books should be possible (yet to try). However, until GUI authors upgrade their programs to accept random castling from polyglot books in FRC/960 chess, the best use for FRC books is for personality books adapted by engine authors.

Trick: If compiling a downloaded windows pgn file, make sure it is in Linux format before attempting to create a book in Linux polyglot. This can be done by loading in nano and re-saving the pgn file to convert from DOS.

An FRC book (and maybe the first one ) can be downloaded on the Schooner website. CCRL_404FRC2012.bin build date is 11/26/2019. It covers all the CCRL FRC 40/4 games since 2012:

https://sites.google.com/site/schoonerchess/downloads

The next version of Schooner will support FRC/960 and the use of FRC polyglot books. It ahould also include a Linux version for the first time. Current Schooner version 2.1 does not support FRC/960 chess.

[hgm]

I do not have permission from the original polyglot author to modify and distribute copies of polyglot.

Polyglot is released under GPL. So everyone has such permission, provided you distribute the source code.

You can also simply use WinBoard or XBoard to make the FRC book. (Or a Xiangqi book, or a Shogi book, or whatever...)

[D Sceviour]

You can also simply use WinBoard or XBoard to make the FRC book. (Or a Xiangqi book, or a Shogi book, or whatever...)

Is the "FRC book" a Polyglot book, or a custom XBoard book?

[hgm]

A Polyglot book. There is no such thing as a custom XBoard book.

[MikeB]

<snip>

Code: Select all

      } else if (my_string_equal(argv[i],"-Chess960")) {
         option_set("Chess960","true");

(3) alter the board_start() to read FRC fen in the book_insert() sub-routine:

Code: Select all

      if (option_get_bool("Chess960")) {
         board_from_fen(board,pgn->fen);
      } else {
         board_start(board);
      }

Bingo!
<snip>

Not quite bingo for me , I had to include 'Option' on the two code snippets above.

} else if (my_string_equal(argv,"-Chess960")) {
option_set(Option,"Chess960","true");

(3) alter the board_start() to read FRC fen in the book_insert() sub-routine:

if (option_get_bool(Option,"Chess960")) {
board_from_fen(board,pgn->fen);
} else {
board_start(board);
}

Otherwise I got this error:

Code: Select all

book_make.c:175:5: error: too few arguments to function 'option_set'
     option_set("Chess960","true");

[hgm]

Also paying attention from the FEN tag should not be dependent on the setting of the UCI_Chess960 option, but on the presence of the tag.

[MikeB]

I created a Chess 960 book based on CCRL and ICCF games and it can be found here at the bottom of the Honey-XR7 release. Although based on over 500,000 games , the average 960 opening position has only 500 games or so. So the book is relatively big, but the lines are short.

https://github.com/MichaelB7/Stockfish/releases/tag/XR7

[D Sceviour]

I created a Chess 960 book based on CCRL and ICCF games and it can be found here at the bottom of the Honey-XR7 release. Although based on over 500,000 games , the average 960 opening position has only 500 games or so. So the book is relatively big, but the lines are short.

https://github.com/MichaelB7/Stockfish/releases/tag/XR7

Thank You!!! The Polyglot 960 book works fine in Schooner. That makes two FRC books now available.

What tool did you use to make this? My changes to Polyglot?

[MikeB]

I created a Chess 960 book based on CCRL and ICCF games and it can be found here at the bottom of the Honey-XR7 release. Although based on over 500,000 games , the average 960 opening position has only 500 games or so. So the book is relatively big, but the lines are short.

https://github.com/MichaelB7/Stockfish/releases/tag/XR7

Thank You!!! The Polyglot 960 book works fine in Schooner. That makes two FRC books now available.

What tool did you use to make this? My changes to Polyglot?

Yes - I used your changes. Thank You!

[D Sceviour]

Not quite bingo for me , I had to include 'Option' on the two code snippets above.

} else if (my_string_equal(argv,"-Chess960")) {
option_set(Option,"Chess960","true");

(3) alter the board_start() to read FRC fen in the book_insert() sub-routine:

if (option_get_bool(Option,"Chess960")) {
board_from_fen(board,pgn->fen);
} else {
board_start(board);
}

Otherwise I got this error:

Code: Select all

book_make.c:175:5: error: too few arguments to function 'option_set'
     option_set("Chess960","true");

The publication date for the Linux source used for Polyglot1.4 was 6/22/2014/2:40 PM. I did not try to look for a later version. I suppose the book_make.cpp file can be shown here if anyone wants to see it.

Code: Select all

// book_make.cpp - modified 11/27/2019 for FRC/960

// includes

#include <cerrno>
#include <cmath>
#include <cstdio>
#include <cstdlib>
#include <cstring>

#include "board.h"
#include "book_make.h"
#include "move.h"
#include "move_do.h"
#include "move_legal.h"
#include "pgn.h"
#include "san.h"
#include "util.h"

#include "option.h" //added 11/27/2019
bool board_from_fen(board_t *, const char *); //added 11/27/2019

// constants

static const int COUNT_MAX = 16384;

static const int NIL = -1;

// types

struct entry_t {
   uint64 key;
   uint16 move;
   uint16 n;
   uint16 sum;
   uint16 colour;
};

struct book_t {
   int size;
   int alloc;
   uint32 mask;
   entry_t * entry;
   sint32 * hash;
};

// variables

static int MaxPly;
static int MinGame;
static double MinScore;
static bool RemoveWhite, RemoveBlack;
static bool Uniform;

static book_t Book[1];

// prototypes

static void   book_clear    ();
static void   book_insert   (const char file_name[]);
static void   book_filter   ();
static void   book_sort     ();
static void   book_save     (const char file_name[]);

static int    find_entry    (const board_t * board, int move);
static void   resize        ();
static void   halve_stats   (uint64 key);

static bool   keep_entry    (int pos);

static int    entry_score    (const entry_t * entry);

static int    key_compare   (const void * p1, const void * p2);

static void   write_integer (FILE * file, int size, uint64 n);

// functions

// book_make()

void book_make(int argc, char * argv[]) {

   int i;
   const char * pgn_file;
   const char * bin_file;

   pgn_file = NULL;
   my_string_set(&pgn_file,"book.pgn");

   bin_file = NULL;
   my_string_set(&bin_file,"book.bin");

   MaxPly = 1024;
   MinGame = 3;
   MinScore = 0.0;
   RemoveWhite = false;
   RemoveBlack = false;
   Uniform = false;

   for (i = 1; i < argc; i++) {

      if (false) {

      } else if (my_string_equal(argv[i],"make-book")) {

         // skip

      } else if (my_string_equal(argv[i],"-pgn")) {

         i++;
         if (argv[i] == NULL) my_fatal("book_make(): missing argument\n");

         my_string_set(&pgn_file,argv[i]);

      } else if (my_string_equal(argv[i],"-bin")) {

         i++;
         if (argv[i] == NULL) my_fatal("book_make(): missing argument\n");

         my_string_set(&bin_file,argv[i]);

      } else if (my_string_equal(argv[i],"-max-ply")) {

         i++;
         if (argv[i] == NULL) my_fatal("book_make(): missing argument\n");

         MaxPly = atoi(argv[i]);
         ASSERT(MaxPly>=0);

      } else if (my_string_equal(argv[i],"-min-game")) {

         i++;
         if (argv[i] == NULL) my_fatal("book_make(): missing argument\n");

         MinGame = atoi(argv[i]);
         ASSERT(MinGame>0);

      } else if (my_string_equal(argv[i],"-min-score")) {

         i++;
         if (argv[i] == NULL) my_fatal("book_make(): missing argument\n");

         MinScore = atof(argv[i]) / 100.0;
         ASSERT(MinScore>=0.0&&MinScore<=1.0);

      } else if (my_string_equal(argv[i],"-only-white")) {

         RemoveWhite = false;
         RemoveBlack = true;

      } else if (my_string_equal(argv[i],"-only-black")) {

         RemoveWhite = true;
         RemoveBlack = false;

      } else if (my_string_equal(argv[i],"-uniform")) {

         Uniform = true;

      } else if (my_string_equal(argv[i],"-Chess960")) {

         option_set("Chess960","true");	//added 11/27/2019

      } else {

         my_fatal("book_make(): unknown option \"%s\"\n",argv[i]);
      }
   }

   book_clear();

   printf("inserting games ...\n");
   book_insert(pgn_file);

   printf("filtering entries ...\n");
   book_filter();

   printf("sorting entries ...\n");
   book_sort();

   printf("saving entries ...\n");
   book_save(bin_file);

   printf("all done!\n");
}

// book_clear()

static void book_clear() {

   int index;

   Book->alloc = 1;
   Book->mask = (Book->alloc * 2) - 1;

   Book->entry = (entry_t *) my_malloc(Book->alloc*sizeof(entry_t));
   Book->size = 0;

   Book->hash = (sint32 *) my_malloc((Book->alloc*2)*sizeof(sint32));
   for (index = 0; index < Book->alloc*2; index++) {
      Book->hash[index] = NIL;
   }
}

// book_insert()

static void book_insert(const char file_name[]) {

   int game_nb;
   pgn_t pgn[1];
   board_t board[1];
   int ply;
   int result;
   char string[256];
   int move;
   int pos;

   ASSERT(file_name!=NULL);

   // init

   game_nb = 0;

   // scan loop

   pgn_open(pgn,file_name);

//added 11/27/2019
   printf("Option Chess960 = %d\n", option_get_bool("Chess960"));

   while (pgn_next_game(pgn)) {

      if (option_get_bool("Chess960")) {
         board_from_fen(board,pgn->fen);	//added 11/27/2019
      } else {
         board_start(board);
      }

      ply = 0;
      result = 0;

      if (false) {
      } else if (my_string_equal(pgn->result,"1-0")) {
         result = +1;
      } else if (my_string_equal(pgn->result,"0-1")) {
         result = -1;
      }

      while (pgn_next_move(pgn,string,256)) {

         if (ply < MaxPly) {

            move = move_from_san(string,board);

            if (move == MoveNone || !move_is_legal(move,board)) {
               my_fatal("book_insert(): illegal move \"%s\" at line %d, column %d\n",string,pgn->move_line,pgn->move_column);
            }

            pos = find_entry(board,move);

            Book->entry[pos].n++;
            Book->entry[pos].sum += result+1;

            if (Book->entry[pos].n >= COUNT_MAX) {
               halve_stats(board->key);
            }

// added 11/27/2019
// dont delete this. Interesting for extended information:

//printf("key 0x%lx\n",(unsigned long) board->key);
//printf(" %d move %s\n",ply,string);

            move_do(board,move);
            ply++;
            result = -result;
         }
      }

      game_nb++;
      if (game_nb % 10000 == 0) printf("%d games ...\n",game_nb);
   }

   pgn_close(pgn);

   printf("%d game%s.\n",game_nb,(game_nb>1)?"s":"");
   printf("%d entries.\n",Book->size);

   return;
}

// book_filter()

static void book_filter() {

   int src, dst;

   // entry loop

   dst = 0;

   for (src = 0; src < Book->size; src++) {
      if (keep_entry(src)) Book->entry[dst++] = Book->entry[src];
   }

   ASSERT(dst>=0&&dst<=Book->size);
   Book->size = dst;

   printf("%d entries.\n",Book->size);
}

// book_sort()

static void book_sort() {

   // sort keys for binary search

   qsort(Book->entry,Book->size,sizeof(entry_t),&key_compare);
}

// book_save()

static void book_save(const char file_name[]) {

   FILE * file;
   int pos;

   ASSERT(file_name!=NULL);

   file = fopen(file_name,"wb");
   if (file == NULL) my_fatal("book_save(): can't open file \"%s\" for writing: %s\n",file_name,strerror(errno));

   // entry loop

   for (pos = 0; pos < Book->size; pos++) {

      ASSERT(keep_entry(pos));

      write_integer(file,8,Book->entry[pos].key);
      write_integer(file,2,Book->entry[pos].move);
      write_integer(file,2,entry_score(&Book->entry[pos]));
      write_integer(file,2,0);
      write_integer(file,2,0);
   }

   fclose(file);
}

// find_entry()

static int find_entry(const board_t * board, int move) {

   uint64 key;
   int index;
   int pos;

   ASSERT(board!=NULL);
   ASSERT(move_is_ok(move));

   ASSERT(move_is_legal(move,board));

   // init

   key = board->key;

   // search

   for (index = key & Book->mask; (pos=Book->hash[index]) != NIL; index = (index+1) & Book->mask) {

      ASSERT(pos>=0&&pos<Book->size);

      if (Book->entry[pos].key == key && Book->entry[pos].move == move) {
         return pos; // found
      }
   }

   // not found

   ASSERT(Book->size<=Book->alloc);

   if (Book->size == Book->alloc) {

      // allocate more memory

      resize();

      for (index = key & Book->mask; Book->hash[index] != NIL; index = (index+1) & Book->mask)
         ;
   }

   // create a new entry

   ASSERT(Book->size<Book->alloc);
   pos = Book->size++;

   Book->entry[pos].key = key;
   Book->entry[pos].move = move;
   Book->entry[pos].n = 0;
   Book->entry[pos].sum = 0;
   Book->entry[pos].colour = board->turn;

   // insert into the hash table

   ASSERT(index>=0&&index<Book->alloc*2);
   ASSERT(Book->hash[index]==NIL);
   Book->hash[index] = pos;

   ASSERT(pos>=0&&pos<Book->size);

   return pos;
}

// resize()

static void resize() {

   int size;
   int pos;
   int index;

   ASSERT(Book->size==Book->alloc);

   Book->alloc *= 2;
   Book->mask = (Book->alloc * 2) - 1;

   size = 0;
   size += Book->alloc * sizeof(entry_t);
   size += (Book->alloc*2) * sizeof(sint32);

   if (size >= 1048576) printf("allocating %gMB ...\n",double(size)/1048576.0);

   // resize arrays

   Book->entry = (entry_t *) my_realloc(Book->entry,Book->alloc*sizeof(entry_t));
   Book->hash = (sint32 *) my_realloc(Book->hash,(Book->alloc*2)*sizeof(sint32));

   // rebuild hash table

   for (index = 0; index < Book->alloc*2; index++) {
      Book->hash[index] = NIL;
   }

   for (pos = 0; pos < Book->size; pos++) {

      for (index = Book->entry[pos].key & Book->mask; Book->hash[index] != NIL; index = (index+1) & Book->mask)
         ;

      ASSERT(index>=0&&index<Book->alloc*2);
      Book->hash[index] = pos;
   }
}

// halve_stats()

static void halve_stats(uint64 key) {

   int index;
   int pos;

   // search

   for (index = key & Book->mask; (pos=Book->hash[index]) != NIL; index = (index+1) & Book->mask) {

      ASSERT(pos>=0&&pos<Book->size);

      if (Book->entry[pos].key == key) {
         Book->entry[pos].n = (Book->entry[pos].n + 1) / 2;
         Book->entry[pos].sum = (Book->entry[pos].sum + 1) / 2;
      }
   }
}

// keep_entry()

static bool keep_entry(int pos) {

   const entry_t * entry;
   int colour;
   double score;

   ASSERT(pos>=0&&pos<Book->size);

   entry = &Book->entry[pos];

   // if (entry->n == 0) return false;
   if (entry->n < MinGame) return false;

   if (entry->sum == 0) return false;

   score = (double(entry->sum) / double(entry->n)) / 2.0;
   ASSERT(score>=0.0&&score<=1.0);

   if (score < MinScore) return false;

   colour = entry->colour;

   if ((RemoveWhite && colour_is_white(colour))
    || (RemoveBlack && colour_is_black(colour))) {
      return false;
   }

   if (entry_score(entry) == 0) return false; // REMOVE ME?

   return true;
}

// entry_score()

static int entry_score(const entry_t * entry) {

   int score;

   ASSERT(entry!=NULL);

   // score = entry->n; // popularity
   score = entry->sum; // "expectancy"

   if (Uniform) score = 1;

   ASSERT(score>=0);

   return score;
}

// key_compare()

static int key_compare(const void * p1, const void * p2) {

   const entry_t * entry_1, * entry_2;

   ASSERT(p1!=NULL);
   ASSERT(p2!=NULL);

   entry_1 = (const entry_t *) p1;
   entry_2 = (const entry_t *) p2;

   if (entry_1->key > entry_2->key) {
      return +1;
   } else if (entry_1->key < entry_2->key) {
      return -1;
   } else {
      return entry_score(entry_2) - entry_score(entry_1); // highest score first
   }
}

// write_integer()

static void write_integer(FILE * file, int size, uint64 n) {

   int i;
   int b;

   ASSERT(file!=NULL);
   ASSERT(size>0&&size<=8);
   ASSERT(size==8||n>>(size*8)==0);

   for (i = size-1; i >= 0; i--) {
      b = (n >> (i*8)) & 0xFF;
      ASSERT(b>=0&&b<256);
      fputc(b,file);
   }
}

// end of book_make.cpp