I would appreciate your thoughts and suggestions for optimization.
The criterions I had in mind when designing this format:
- Fast import/export of a large number of games
- Instant lookup up certain positions
- Compactness, compared to other formats that feature instant position lookup
- Slower than other formats in exporting individual games
- Slower than other formats in (fully) deleting individual games
- Slower than other formats in answering questions like: “How often does Player x castle queenside?”
- Game Metadata
- Position-Table
- Endgame-Table
One line per game; each line holds the data from the pgn tags (players, result, elo, etc.).
Additionally it stores an Opening ID. This ID references into a general table with the most common opening lines.
It stores the number of moves in the game.
It stores the position of where the game converted into a 5-men ending or the final position if the game terminated earlier.
It stores a game-hash. The game hash is generated by xoring every move ror(hash(move), ply)
Position-Table
The position table is a collection of zobrist keys of positions that have been reached by any game in the database between leaving the opening table and entering the 5-men ending.
Endgame-Table
It holds a collection of move-lines of games that go beyond a 5 men-ending. It references into the Game Metadata table.
Additional tables might be needed for variation-pointers and comments.
Use-case 1: Find all games that feature a certain position:
- Check if the position is present in the Position-Table
- Check if an entry from the Game Metadata table points to the position
- Generate all moves from the current position
- For each move continue at 1 (guard against endless loops)
- Load the ending from the Endgame-Table
- Start with the final position
- Check if the position is present in the Position Table
- Check if the position equals the opening position
- Check if the game hash is correct
- Generate all retro-moves from the current position
- For each move continue at 3 (guard against endless loops)