Opening book for chess variants

[Evert]

One of the things I'd like to add to SjaakII is support for opening books. This will require a bit of meta-data on the engine side, which is ok. Getting games to derive opening books from is a bit more tricky, but I can always sink computer time into generating (relatively) high-quality games for some variants.

The real question I have is with respect to book format. I've looked at the polyglot format (http://hgm.nubati.net/book_format.html) which seems ok, but it has a few limitations: first of all it doesn't really specify what variant the book is intended for, it just mangles the position keys. I'm not really happy with that, I'd like to be able to read the variant from a header. It only supports up to 24 piece types, which is mostly ok but SjaakII supports up to 32. In short, it'll do but it doesn't seem optimal for my purpose. Before I go and reinvent the wheel though, I would like to ask whether anyone actually uses polyglot for chess variants, and whether any opening books are available somewhere?

[hgm]

I already relaxed the restriction on the number of pieces in XBoard, for the purpose of doing Chu Shogi books. There also is a method for encoding Lion double moves.

The variant is part of the position key, to exclude there can be any confusion on which variant a position belongs to, to make milti-variant books possible. These are for instance needed when you play a variant engine on an ICS, where you don't know in advance for what variant you will be challenged. But nothing prevents you from storing only positions for a single variant in a book.

Note there is a proposal for incorporating a header in a Polyglot book, by Michel van den Bergh. XBoard does not support that yet, though. Mainly because it would not be interested anyway to see what is in the header; author and copyright info in there can be nice, but I did not have any intention to let XBoard display that. Of course you can also use a naming convention for books that would reveal the variant, (e.g. bigbook2.xiangqi.bk) and let Sjaak read it from the filename instead of a the file contents.

Anyway, this is the current encoding of the board part used by XBoard:

Code: Select all

            if(p != EmptySquare){
		    int j = (int)p, promoted = 0;
		    j -= (j >= (int)BlackPawn) ? (int)BlackPawn :(int)WhitePawn;
		    if(j >= CHUPROMOTED WhitePawn) promoted++, j -= CHUPROMOTED WhitePawn;
		    if(j > (int)WhiteQueen) j++;  // make space for King
		    if(j > (int) WhiteKing) j = (int)WhiteQueen + 1;
		    p_enc = 2*j + ((int)p < (int)BlackPawn);
		    // holdings squares get numbers immediately after board; first left, then right holdings
		    if(f == BOARD_LEFT-2) squareNr = (BOARD_RGHT - BOARD_LEFT)*BOARD_HEIGHT + r; else
		    if(f == BOARD_RGHT+1) squareNr = (BOARD_RGHT - BOARD_LEFT + 1)*BOARD_HEIGHT + r; else
		    squareNr = (BOARD_RGHT - BOARD_LEFT)*r + (f - BOARD_LEFT);
		    // note that in normal Chess squareNr < 64 and p_enc < 12. The following code
		    // maps other pieces and squares in this range, and then modify the corresponding
		    // Zobrist random by rotating its bitpattern according to what the piece really was.
		    pieceGroup = p_enc / 12;
		    p_enc      = p_enc % 12;
		    Zobrist = RandomPiece[64*p_enc + (squareNr & 63)];
		    if(pieceGroup & 4) Zobrist *= 987654321;
		    switch(pieceGroup & 3) {
			case 1: // pieces 5-10 (FEACWM)
				Zobrist = (Zobrist << 16) ^ (Zobrist >> 48);
				break;
			case 2: // pieces 11-16 (OHIJGD)
				Zobrist = (Zobrist << 32) ^ (Zobrist >> 32);
				break;
			case 3: // pieces 17-20 (VLSU)
				Zobrist = (Zobrist << 48) ^ (Zobrist >> 16);
				break;
		    }
		    if(promoted) Zobrist ^= 123456789*RandomPiece[squareNr & 63];
		    if(squareNr &  64) Zobrist = (Zobrist << 8) ^ (Zobrist >> 56);
		    if(squareNr & 128) Zobrist = (Zobrist << 4) ^ (Zobrist >> 60);
		    // holdings have separate (additive) key, to encode presence of multiple pieces on same square
		    if(f == BOARD_LEFT-2) holdingsKey += Zobrist * boards[moveNr][r][f+1]; else
		    if(f == BOARD_RGHT+1) holdingsKey += Zobrist * boards[moveNr][r][f-1]; else
                key ^= Zobrist;
            }
        }
    }

[hgm]

I guess this code is not exactly what I intended, as 'CHUPROMOTED x' would mean x+11 in other variants than Chu, and x+22 in Chu. (Which in the development version now already has grown to x+27.) I should have written WhiteTokin instead of CHUPROMOTED WhitePawn, which would mean the same in any variant.

I see there is a problem with the new pieces I added at the end of the chu-unpromoted series (after Unicorn), which drives up the number of pieces there to over 24. So currently only Lion and Wolf would have an encoding, and Camel, Zebra, Wizard and Amazon present a problem. I guess I will have to adopt a rule for creating keys for unpromoted pieces 24-47.

Anyway, the current limitation is 24 unpromoted piece types, 24 promoted piece types and 192 board squares. But it can always be extended when needed.

[hgm]

Oops, I missed this: there already is a provision for unpromoted pieces 24-47 in the above code. The if(pieceGroup & 4) in the code above takes care of that; it multiplies the base key by a factor for peiecGroups 4-7, and every group is 6 pievce types (x 2 colors)

[Evert]

You wouldn't happen to have a table that maps piece-type to polyglot piece-number for defined variants, would you?

I'm tempted to go for my own polyglot-like book format that includes some meta-data and a conversion command; thing is, if my own format doesn't offer that much extra it would be better to just adopt an existing standard...

[hgm]

You wouldn't happen to have a table that maps piece-type to polyglot piece-number for defined variants, would you?

Basically that is just the pieceToCharTable for that variant. Except that the King is moved up to 6th place, instead of last, to remain compatible with the original Chess format. And that the number then is multiplied by two, and 1 is added for white. So the Polyglot order is

pPnNbBrRqQkKfFeEaAcCwWmMoOhHiIjJgGdDvVlLsSuU

I'm tempted to go for my own polyglot-like book format that includes some meta-data and a conversion command; thing is, if my own format doesn't offer that much extra it would be better to just adopt an existing standard...

Michel's header proposal just uses the non-key part of entries with key 0 to store header info. There could be all kind of info the header, also something that would redefine the hash mapping or the function of the fields.

The main problem with the existing format is that the move field is just 16 bits. That limits the board size to 16x16 boards in the (fromSqr, toSqr) system, and there would be no bits left to indicate promotions. A (fromSqr, orientation, distance) encoding would be slightly more compact, however. The 25x25 board of Tai Shogi has 625 square numbers, which leaves 104.85 moves for each square. With 4 ray orientations and 24 distances along them only 96 would be needed. (Promotion is mandatory and implied in Tai.) The remaining 8 could be Knight jumps. No codes left for Lion double-moves however. Of course not all squares have diagonals of length 24. I am not sure I would ever want to make a book for Tai Shogi, though...

Of course we could hijack part of the learn fields on bigger boards, to create more room for the move. 24 bits might be enough.