Tablebase suggestion

Discussion of anything and everything relating to chess playing software and machines.

Moderator: Ras

syzygy
Posts: 5999
Joined: Tue Feb 28, 2012 11:56 pm

Re: Tablebase suggestion

Post by syzygy »

It also seems to work for chess and more than 2 piece types:

Code: Select all

from symmetry_indexer import SymmetryIndexer

def sq_index(r, c):
    return 8 * r + c

def sq_coord(i):
    return divmod(i, 8)

def make_chess_D4():
    transforms = [
        lambda r, c: (r, c),              # identity
        lambda r, c: (c, 7 - r),          # rotate 90
        lambda r, c: (7 - r, 7 - c),      # rotate 180
        lambda r, c: (7 - c, r),          # rotate 270
        lambda r, c: (r, 7 - c),          # vertical mirror
        lambda r, c: (7 - r, c),          # horizontal mirror
        lambda r, c: (c, r),              # main diagonal
        lambda r, c: (7 - c, 7 - r),      # anti-diagonal
    ]

    perms = []

    for f in transforms:
        p = [0] * 64
        for i in range(64):
            r, c = sq_coord(i)
            rr, cc = f(r, c)
            p[i] = sq_index(rr, cc)
        perms.append(p)

    return perms


if __name__ == "__main__":
    # colors:
    # 0 = empty
    # 1 = white piece
    # 2 = black piece
    idx = SymmetryIndexer(
        n_squares=64,
        perms=make_chess_D4(),
        n_colors=4,
    )

    print("orbit sizes:", [len(b["orbit"]) for b in idx.blocks])
    print("orbits:")
    for b in idx.blocks:
        print(b["orbit"])

    total = idx.count(target_counts=(1, 1, 2))
    print("count for 1+1+2 pieces:", total)

    pos = [0] * 64
    pos[sq_index(0, 0)] = 1
    pos[sq_index(1, 2)] = 2
    pos[sq_index(4, 5)] = 3
    pos[sq_index(7, 7)] = 3

    r = idx.rank(pos, target_counts=(1, 1, 2))
    print("rank:", r)

    pos2 = idx.unrank(r, target_counts=(1, 1, 2))
    print("unranked:", pos2)

    print("roundtrip ok:", tuple(idx.canonicalize(pos)) == pos2)
So (1,1,2) corresponds to a KNNvK (where the Ks may be on adjacent squares):

Code: Select all

$ python example_chess.py 
orbit sizes: [4, 4, 4, 4, 8, 8, 8, 8, 8, 8]
orbits:
[0, 7, 56, 63]
[9, 14, 49, 54]
[18, 21, 42, 45]
[27, 28, 35, 36]
[1, 6, 8, 15, 48, 55, 57, 62]
[2, 5, 16, 23, 40, 47, 58, 61]
[3, 4, 24, 31, 32, 39, 59, 60]
[10, 13, 17, 22, 41, 46, 50, 53]
[11, 12, 25, 30, 33, 38, 51, 52]
[19, 20, 26, 29, 34, 37, 43, 44]
count for 1+1+2 pieces: 953666
rank: 949116
unranked: (0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0)
roundtrip ok: True
953666 is correct. The rank 949116 is a bit surprisingly close to that number, perhaps because (0,0) is one of the squares.

Unforunately picking different squares returns an error. There seems to be a bug, which ChatGPT now claims to have fixed. I'll try again tomorrow.
Rein Halbersma
Posts: 779
Joined: Tue May 22, 2007 11:13 am

Re: Tablebase suggestion

Post by Rein Halbersma »

The original post here contained implementation details (stabilizer specific subdivisions) that can be eliminated without loss of generality. ChatGPT is a bit of an idiot savant, sometimes it will pick really weird column-order coordinates or opaquely named data structures, or not immediately see simplifications. But it will plough through all that and exhaustively test everything. 80% of the effort is rephrasing prompts. These AI tools have come a long way.
syzygy
Posts: 5999
Joined: Tue Feb 28, 2012 11:56 pm

Re: Tablebase suggestion

Post by syzygy »

Rein Halbersma wrote: Wed May 20, 2026 11:41 pm The original post here contained implementation details (stabilizer specific subdivisions) that can be eliminated without loss of generality. ChatGPT is a bit of an idiot savant, sometimes it will pick really weird column-order coordinates or opaquely named data structures, or not immediately see simplifications. But it will plough through all that and exhaustively test everything. 80% of the effort is rephrasing prompts. These AI tools have come a long way.
Yes, it is amazing what is possible now. AI is going to accelerate progress in many if not all fields.

Btw, I knew we had already discussed this problem in the past, and it turns out to be almost exactly 10 years ago:
viewtopic.php?p=671955#p671955

We just had to wait for AI to come along :)

I still have to find some time to try to fully understand this approach (and your paper).
Rein Halbersma
Posts: 779
Joined: Tue May 22, 2007 11:13 am

Re: Tablebase suggestion

Post by Rein Halbersma »

So the best way to summarize the approach I have so far is this:

A binomial table builder recurses over simply putting a piece on a square or not, as

binom(n, k) = sum_{all ways to put 0..1 pieces onto the next square} binom(n - 1, k - # pieces on that square)
= binom(n -1, k) + binom(n-1, k-1),

where the sum expand just into two terms, corresponding to placing 0 or 1 piece onto the next square. This is simply Pascal's triangle identity.

For draughts, the "canomial" builder recurses over all ways to put up to 4 pieces into n remaining orbits, ordering any particular configuration of the next orbit into the canonical way (i.e. sorted list of squares lexicographically least order), and tracking which non-trivial symmetries were used to put it into canonical order, and then recursing. In formula:

canom(n, k, 0b111) = sum_{all ways to put 0..4 pieces into the next orbit} canom(n - 1, k - #pieces in that orbit, 0b111 & ~symmetries used to put that new orbit into canonical order)

where the & ~ is just bitwise minus for removing the used symmetry bits from the mask. (Remember in C++ you can write 0b as prefix for a bit pattern). So the recursion builds a table with an extra index, which is the bitmask.

For draughts this is just a 3-bit mask, so the extra index ranges over 8 possibilities. For chess it's 7-bit mask, so 128 possibilities. This is why it doesn't scale for games with huge symmetries. Nine Men's Morris e.g. has a 16-fold symmetry so you have 2^15 = 32K possible remaining symmetries. And the board game Pentago has 2048 fold symmetry so you have 2^2047 possible remaining symmetries. So for those games, this approach either is not saving very much due to the enormous overhead of the helper tables, or simply impossible to compute.
syzygy
Posts: 5999
Joined: Tue Feb 28, 2012 11:56 pm

Re: Tablebase suggestion

Post by syzygy »

Rein Halbersma wrote: Fri May 22, 2026 9:12 am So the best way to summarize the approach I have so far is this:

A binomial table builder recurses over simply putting a piece on a square or not, as

binom(n, k) = sum_{all ways to put 0..1 pieces onto the next square} binom(n - 1, k - # pieces on that square)
= binom(n -1, k) + binom(n-1, k-1),

where the sum expand just into two terms, corresponding to placing 0 or 1 piece onto the next square. This is simply Pascal's triangle identity.
So if you start by looking at square 63, and you let "empty" choose the range 0..binom(n-1,k)-1, then filling square 63 with the kth piece means you add binom(63,k) to the index. Same for square 62,61,...,0, and you recover the combinadics approach to ranking/unranking.
For draughts, the "canomial" builder recurses over all ways to put up to 4 pieces into n remaining orbits, ordering any particular configuration of the next orbit into the canonical way (i.e. sorted list of squares lexicographically least order), and tracking which non-trivial symmetries were used to put it into canonical order, and then recursing. In formula:
So indeed you walk over the orbits of squares of the board. If the group of symmetries is the trivial group, you end up with the binomial case discussed above.

Filling an orbit in the canonical way will usually take out some (and probably quite often all) of the remaining symmetries. I guess you could then determine the remaining orbits of the squares? If no symmetry is left, you end up with the easy binomial case. This is basically what I did in my manually coded ranking/unranking functions.

Of course there is no reduction in the remaining symmetries if an orbit stays empty. I guess this can be used to treat similar orbits (with same stabilizer) together, i.e. order them and consider the lowest-ordered orbit that receives a piece. (Perhaps your program does this.)
canom(n, k, 0b111) = sum_{all ways to put 0..4 pieces into the next orbit} canom(n - 1, k - #pieces in that orbit, 0b111 & ~symmetries used to put that new orbit into canonical order)

where the & ~ is just bitwise minus for removing the used symmetry bits from the mask. (Remember in C++ you can write 0b as prefix for a bit pattern). So the recursion builds a table with an extra index, which is the bitmask.
I guess I understand this, but I would have to think about how to then build the table and get everything right.
The 'n' in canom(n, k, 0b111) pins down the orbit being filled, right?
So the main trick is that you can forget about what happened before; only the value of k and the bitmask matter.
For draughts this is just a 3-bit mask, so the extra index ranges over 8 possibilities. For chess it's 7-bit mask, so 128 possibilities. This is why it doesn't scale for games with huge symmetries. Nine Men's Morris e.g. has a 16-fold symmetry so you have 2^15 = 32K possible remaining symmetries. And the board game Pentago has 2048 fold symmetry so you have 2^2047 possible remaining symmetries. So for those games, this approach either is not saving very much due to the enormous overhead of the helper tables, or simply impossible to compute.
You can improve on 2^|G| by considering the subgroups of the symmetry group G instead, but your table builder will probably discover this by itself.

D_4 has 9 subgroups, so 10 possible groups of remaining symmetries. Each of them can occur as the stabilizer of a subset of an orbit.
Nine Men's Morris seems to have 19 possible groups of remaining symmetries.

Maybe I will try to write a program that generates ranking/unranking functions for <= N pieces in a non-table style. Not to use in my generator but just to finally do what I would have liked to do automatically in 2005.
syzygy
Posts: 5999
Joined: Tue Feb 28, 2012 11:56 pm

Re: Tablebase suggestion

Post by syzygy »

syzygy wrote: Sat May 23, 2026 1:16 amMaybe I will try to write a program that generates ranking/unranking functions for <= N pieces in a non-table style.
Let's first see how it works for chess in more detail.

We start with a particular combination of white and black pieces (no pawns). What matters is not so much the pieces themselves but their multiplicities.

For regular chess we know that we have two kings with multiplicity 1. We can start by placing them in the usual way. This also avoids a lot of illegal positions.
Then in 441 out of 462 cases we're already done: no symmetries left, and the remaining pieces can be placed group by group using combinadics.

In the remaining 21 cases, both kings are on the main diagonal. There is now exactly one diagonal symmetry left.
There are 28 orbits of size 2, 6 orbits of size 1.

Now what I would normally do (to obtain a perfect indexing function) is order the pieces, with pieces of multiplicity one going first.
If the first piece of multiplicity 1 is not on the diagonal, then we place it in the square of its orbit that is below the diagonal (and encode the orbit in the index). This uses up the last symmetry and the rest is easy.
If it is on the diagonal, then we place it there (encoding the orbit in the index) and we move on to the next piece.
If the first (or next) piece has multiplicity 2, then we check whether both are in the same orbit. If not, we place the piece in the lowest orbit (encoding the orbit in the index), and the rest is easy. If they are in the same orbit, we place them there, encode that orbit in the index, and move on to the next piece. (And then there is the further case of both being in 1-square orbits on the diagonal.)
If the first (or next) piece has multiplicity 3, then we check how many are in the lowest orbit, etc.
As for "the rest is easy", any remaining pieces in the current group of pieces will have to be placed in the squares which formed the orbits ranking higher than the orbit that received 1 piece. The remaining groups of pieces can be placed in the normal way. With pext/pdep this has all become very easy.

So my "normal" approach is more per-piece-group than per-orbit (but then first looks at the lowest orbit that is touched by the piece group).
I think it works because at each step either you fill a complete orbit or you remove the last symmetry.
It seems possible to create a general function for it that uses precomputed tables similar to yours for encoding the orbits and squares on which pieces are placed into the index. The relevant state information is (#empty orbits of size 2, #empty orbits of size 1, multiplicity signature of the remaining pieces to be placed).

This also works for variants such as atomic chess, where the kings may touch but you still will always have exactly two kings of different colors.

Interestingly there is no need to order the piece groups in order of increasing multiplicity. Allowing both (1,2) and (2,1) does increase table size, but that's it. The final index range stays the same (of course). With my old approach of having separate functions for each multiplicity signature, it just didn't make sense to have two functions rank_12() and rank_21() instead of one. But the disadvantage of having to sort by multiplicity was that flexibility in piece ordering was lost, and it turns out that some piece orderings compress much better than others. This is why I concluded it was better not to bother with perfect indexing and to just accept a small percentage of incompressible redundancy (the piece-ordering flexiblity more than making up for it). [I even went further and did not place the two Ks together at the cost of including many more illegal positions in the index range, which however can be compressed away quite well.]
So there's probably no point in trying to get rid of the tables by transforming the method into an inflexible series of if-then statements and index calculations.

My new generator compresses pawnless 8-piece tables as 462 separate tables corresponding to KK placements. For 441 of them there is nothing to gain (already perfect), but the remaining 21 have about 50% incompressible redundancy. This table-based approach might be simple enough to fully remove that redundancy (while preserving ordering flexibility).

This (extended) "normal" approach is not quite your approach (but the state idea is also what makes this work).

My approach starts to get into trouble when the orbits are larger.
In suicide chess, the king is a normal piece and you can have none or multiple white or black kings. So there is no guaranteed easy reduction to a position with at most one diagonal symmetry left. So now we have to start with 6 orbits of size 8 and 4 orbits of size 4. And if we place pieces group by group, then we get partially filled orbits, which means we get more states to track (and also symmetries of course).

I think it can still be made to work, though. If we partially fill an orbit, then it seems the remaining squares in the orbit necessarily form new orbits under the group of remaining symmetries. So after losing a symmetry it seems we should just look at the orbits that remain under the smaller symmetry group. I think we then only need to track the number of orbits per "orbit type". I still have to think about the proper definition of an "orbit type". In general the size of the orbit is probably not enough to define the type, but for the chess board perhaps it is. Even if there are more, there are probably just a few types, so keeping track of them as part of the state should not explode state space.
(There may be complications that I am overlooking here.)

Your approach looks at orbits first. Filling/coloring an orbit takes out that orbit completely and leaves the other orbits untouched. So no need to track partially-filled orbits. But you have more numbers to store per state: one for each coloring, and with (suicide) chess you need up to N+1 colors for N pieces (of course then you won't need to cover all 8^(N+1) possible ways to color the 8 squares of an orbit).
syzygy
Posts: 5999
Joined: Tue Feb 28, 2012 11:56 pm

Re: Tablebase suggestion

Post by syzygy »

syzygy wrote: Sat May 23, 2026 1:16 amYou can improve on 2^|G| by considering the subgroups of the symmetry group G instead, but your table builder will probably discover this by itself.

D_4 has 9 subgroups, so 10 possible groups of remaining symmetries. Each of them can occur as the stabilizer of a subset of an orbit.
Nine Men's Morris seems to have 19 possible groups of remaining symmetries.
I suppose conjugate subgroups must give the same counts, which should help to reduce 10 to 8 and 19 to 11. (I think the corresponding bitmasks will have identical tables associated to them.)
https://groupprops.subwiki.org/wiki/Sub ... l_group:D8
https://groupprops.subwiki.org/wiki/Sub ... _group:D16
Perhaps a transformation would be needed to make it work correctly (not sure).
syzygy
Posts: 5999
Joined: Tue Feb 28, 2012 11:56 pm

Re: Tablebase suggestion

Post by syzygy »

syzygy wrote: Sat May 23, 2026 4:14 pmMy approach starts to get into trouble when the orbits are larger.
In suicide chess, the king is a normal piece and you can have none or multiple white or black kings. So there is no guaranteed easy reduction to a position with at most one diagonal symmetry left. So now we have to start with 6 orbits of size 8 and 4 orbits of size 4. And if we place pieces group by group, then we get partially filled orbits, which means we get more states to track (and also symmetries of course).

I think it can still be made to work, though. If we partially fill an orbit, then it seems the remaining squares in the orbit necessarily form new orbits under the group of remaining symmetries. So after losing a symmetry it seems we should just look at the orbits that remain under the smaller symmetry group. I think we then only need to track the number of orbits per "orbit type". I still have to think about the proper definition of an "orbit type". In general the size of the orbit is probably not enough to define the type, but for the chess board perhaps it is. Even if there are more, there are probably just a few types, so keeping track of them as part of the state should not explode state space.
(There may be complications that I am overlooking here.)
This approach also seems to work.
ChatGPT also pointed out that there is more about "orbit type" than its size:
The main subtlety is this:

Orbits of the same size under H are not always equivalent as H-sets relative to the remaining larger symmetry structure.

For pure counting from a fixed stabilizer H, two free H-orbits of size 2 may be interchangeable if no further geometric distinction matters. But for canonicalization and for determining future stabilizers, you may need to know how those orbits transform under elements of the normalizer or under possible residual symmetries after later placements. So a robust implementation should classify remaining orbits by their actual H-action type and possibly their embedding in the board, not merely by size.

A safer DP state is therefore:

(H, c_1, ..., c_m, λ)

where c_i counts remaining empty orbits of each precomputed orbit type for subgroup H.

For chess D_4, the number of subgroups and orbit types is small, so this is very feasible.
I think there should also be an indication in the state of where placement of the group of same-type pieces currently being placed should continue. I'm not sure yet how ChatGPT deals with this in the same Python code it generated:

Code: Select all

from __future__ import annotations

from dataclasses import dataclass
from functools import lru_cache
from itertools import combinations
from typing import Callable, Iterable, Sequence
from random import randrange

Mask = int
Perm = tuple[int, ...]


def bits(mask: Mask) -> list[int]:
    """Return set-bit indices in increasing order."""
    out: list[int] = []
    while mask:
        lsb = mask & -mask
        out.append(lsb.bit_length() - 1)
        mask ^= lsb
    return out


def bit_count(mask: Mask) -> int:
    return mask.bit_count()


def mask_from_squares(squares: Iterable[int]) -> Mask:
    m = 0
    for s in squares:
        m |= 1 << s
    return m


def mask_key(mask: Mask) -> tuple[int, ...]:
    """Lexicographic order on square lists, not merely integer order."""
    return tuple(bits(mask))


@dataclass(frozen=True)
class FiniteBoardAction:
    """A finite board together with a finite permutation group acting on squares.

    The group is represented explicitly as permutations of range(num_squares).
    The first permutation is expected to be the identity. The implementation does
    not require a multiplication table; stabilizers are represented as tuples of
    indices into self.perms.

    For speed, the constructor precomputes the image of each singleton bit under
    each permutation. Then applying a permutation to a mask is only a loop over
    the set bits, OR'ing precomputed singleton images.
    """

    num_squares: int
    perms: tuple[Perm, ...]

    def __post_init__(self) -> None:
        if not self.perms:
            raise ValueError("at least the identity permutation is required")
        n = self.num_squares
        for p in self.perms:
            if len(p) != n or sorted(p) != list(range(n)):
                raise ValueError("invalid permutation")
        if self.perms[0] != tuple(range(n)):
            raise ValueError("perms[0] must be the identity")

        bit_image = tuple(
            tuple(1 << p[s] for s in range(n))
            for p in self.perms
        )
        object.__setattr__(self, "bit_image", bit_image)

    @property
    def full_group(self) -> tuple[int, ...]:
        return tuple(range(len(self.perms)))

    @lru_cache(maxsize=None)
    def apply_mask(self, perm_index: int, mask: Mask) -> Mask:
        img = self.bit_image[perm_index]
        out = 0
        x = mask
        while x:
            lsb = x & -x
            s = lsb.bit_length() - 1
            out |= img[s]
            x ^= lsb
        return out

    @lru_cache(maxsize=None)
    def mask_key_cached(self, mask: Mask) -> tuple[int, ...]:
        return mask_key(mask)

    @lru_cache(maxsize=None)
    def canonical_mask(self, mask: Mask, subgroup: tuple[int, ...]) -> Mask:
        return min((self.apply_mask(g, mask) for g in subgroup), key=self.mask_key_cached)

    @lru_cache(maxsize=None)
    def stabilizer_of_mask(self, mask: Mask, subgroup: tuple[int, ...]) -> tuple[int, ...]:
        return tuple(g for g in subgroup if self.apply_mask(g, mask) == mask)

    def transform_position(self, masks: tuple[Mask, ...], g: int) -> tuple[Mask, ...]:
        return tuple(self.apply_mask(g, m) for m in masks)

    def canonical_position(self, masks: tuple[Mask, ...]) -> tuple[Mask, ...]:
        return min((self.transform_position(masks, g) for g in self.full_group),
                   key=lambda ms: tuple(self.mask_key_cached(m) for m in ms))


def chess_d4_action() -> FiniteBoardAction:
    """Return the standard D4 action on an 8x8 chessboard.

    Square numbering is row-major: square = 8 * rank + file, with rank/file in
    0..7. The particular orientation does not matter as long as rank() and
    unrank() use the same convention.
    """

    def sq(f: int, r: int) -> int:
        return 8 * r + f

    transforms: list[Callable[[int, int], tuple[int, int]]] = [
        lambda f, r: (f, r),          # identity
        lambda f, r: (7 - r, f),      # rotate 90
        lambda f, r: (7 - f, 7 - r),  # rotate 180
        lambda f, r: (r, 7 - f),      # rotate 270
        lambda f, r: (7 - f, r),      # mirror vertical axis
        lambda f, r: (f, 7 - r),      # mirror horizontal axis
        lambda f, r: (r, f),          # mirror main diagonal
        lambda f, r: (7 - r, 7 - f),  # mirror anti-diagonal
    ]

    perms: list[Perm] = []
    for tr in transforms:
        p = [0] * 64
        for r in range(8):
            for f in range(8):
                nf, nr = tr(f, r)
                p[sq(f, r)] = sq(nf, nr)
        perms.append(tuple(p))
    return FiniteBoardAction(64, tuple(perms))


@dataclass(eq=False)
class SymmetryRanker:
    """Exact symmetry-reduced ranker/unranker for fixed material.

    material is an ordered sequence of multiplicities, one entry per piece type.
    Example: [1, 1, 2, 1] could mean WK, BK, two white rooks, one black bishop.

    Kings are not special. Pawns are not special either; this class knows only
    anonymous square-occupying types. Legality constraints such as adjacent kings,
    check, side to move, castling, en passant, etc. are intentionally outside the
    model.

    The algorithm ranks orbits of positions under board.action.full_group.
    A position is represented as tuple[Mask, ...], one mask per material group.
    The masks must be pairwise disjoint and have the specified popcounts.
    """

    action: FiniteBoardAction
    material: tuple[int, ...]

    def __init__(self, action: FiniteBoardAction, material: Sequence[int]):
        self.action = action
        self.material = tuple(material)
        if any(m < 0 for m in self.material):
            raise ValueError("negative multiplicity")
        if sum(self.material) > action.num_squares:
            raise ValueError("too many pieces for board")
        self.all_squares_mask = (1 << action.num_squares) - 1

    def validate(self, masks: tuple[Mask, ...]) -> None:
        if len(masks) != len(self.material):
            raise ValueError("wrong number of piece groups")
        occ = 0
        for i, (m, need) in enumerate(zip(masks, self.material)):
            if m & ~self.all_squares_mask:
                raise ValueError(f"piece group {i} has squares outside the board")
            if bit_count(m) != need:
                raise ValueError(f"piece group {i} has popcount {bit_count(m)}, expected {need}")
            if occ & m:
                raise ValueError("overlapping pieces")
            occ |= m

    @lru_cache(maxsize=None)
    def _canonical_choices(self, subgroup: tuple[int, ...], available: Mask, count: int) -> tuple[Mask, ...]:
        """Canonical representatives of count-subsets of available squares under subgroup.

        This is the exact but simple part of the implementation. It enumerates
        combinations of currently available squares and keeps one representative
        per subgroup-orbit. For high multiplicities this should be replaced by
        the orbit-inventory DP discussed in the design notes.
        """
        return tuple(row[0] for row in self._choice_rows(subgroup, available, count))

    @lru_cache(maxsize=None)
    def _choice_rows(self, subgroup: tuple[int, ...], available: Mask, count: int) -> tuple[tuple[Mask, tuple[int, ...]], ...]:
        """Precomputed rows (choice, stabilizer) for one recursive state.

        rank(), unrank(), canonicalize(), and count() all need the stabilizer of
        each admissible canonical choice. Keeping it here avoids recomputing it
        in each caller.
        """
        if count == 0:
            return ((0, subgroup),)

        sqs = bits(available)
        seen: set[Mask] = set()
        reps: list[Mask] = []
        for comb in combinations(sqs, count):
            m = mask_from_squares(comb)
            if m in seen:
                continue
            orbit = tuple(self.action.apply_mask(g, m) for g in subgroup)
            seen.update(orbit)
            rep = min(orbit, key=self.action.mask_key_cached)
            reps.append(rep)

        reps.sort(key=self.action.mask_key_cached)
        return tuple((rep, self.action.stabilizer_of_mask(rep, subgroup)) for rep in reps)

    def precompute(self) -> int:
        """Force construction of the reachable count/choice tables.

        This is useful before timing rank()/unrank(). It returns the total number
        of symmetry-distinct placements, i.e. the same value as count().
        """
        return self.count()


    @lru_cache(maxsize=None)
    def count(self, group_index: int = 0, occupied: Mask = 0,
              subgroup: tuple[int, ...] | None = None) -> int:
        """Number of symmetry-distinct completions from the given recursive state."""
        if subgroup is None:
            subgroup = self.action.full_group
        if group_index == len(self.material):
            return 1

        need = self.material[group_index]
        available = self.all_squares_mask & ~occupied
        total = 0
        for choice, new_subgroup in self._choice_rows(subgroup, available, need):
            total += self.count(group_index + 1, occupied | choice, new_subgroup)
        return total

    def canonicalize(self, masks: tuple[Mask, ...]) -> tuple[Mask, ...]:
        """Canonicalize using the same recursive rule as rank()/unrank().

        A globally lexicographic representative of the full G-orbit need not be
        canonical at every recursive stabilizer step. Therefore canonicalization
        must be recursive, not just min(g * position) over the full group.
        """
        self.validate(masks)
        return self._canonicalize_from(0, 0, self.action.full_group, masks)

    def _canonicalize_from(self, group_index: int, occupied: Mask,
                           subgroup: tuple[int, ...], masks: tuple[Mask, ...]) -> tuple[Mask, ...]:
        if group_index == len(self.material):
            return ()

        current = masks[group_index]
        available = self.all_squares_mask & ~occupied
        orbit = {self.action.apply_mask(g, current) for g in subgroup}

        best: tuple[Mask, ...] | None = None
        for choice, new_subgroup in self._choice_rows(subgroup, available, self.material[group_index]):
            if choice not in orbit:
                continue

            # Pick a subgroup element that maps current to this canonical choice,
            # and apply it to all later groups before recursing.
            for g in subgroup:
                if self.action.apply_mask(g, current) == choice:
                    transformed = tuple(
                        masks[i] if i <= group_index else self.action.apply_mask(g, masks[i])
                        for i in range(len(masks))
                    )
                    tail = self._canonicalize_from(group_index + 1, occupied | choice,
                                                   new_subgroup, transformed)
                    candidate = (choice,) + tail
                    if best is None or tuple(mask_key(m) for m in candidate) < tuple(mask_key(m) for m in best):
                        best = candidate
                    break

        if best is None:
            raise ValueError("position is not representable in the current recursive state")
        return best

    def rank(self, masks: tuple[Mask, ...]) -> int:
        """Rank the symmetry orbit containing masks.

        Non-canonical input is accepted: it is first canonicalized according to
        the same recursive canonical path used by unrank(). The returned rank is
        in 0..count()-1.
        """
        masks = self.canonicalize(masks)
        rank = 0
        occupied = 0
        subgroup = self.action.full_group

        for gi, current in enumerate(masks):
            need = self.material[gi]
            available = self.all_squares_mask & ~occupied
            if current & ~available or bit_count(current) != need:
                raise ValueError("invalid canonical position")

            choices = self._choice_rows(subgroup, available, need)
            found = False
            for choice, new_subgroup in choices:
                block = self.count(gi + 1, occupied | choice, new_subgroup)
                if choice == current:
                    occupied |= choice
                    subgroup = new_subgroup
                    found = True
                    break
                rank += block
            if not found:
                raise ValueError("position is not represented by this ranker state")

        return rank

    def unrank(self, index: int) -> tuple[Mask, ...]:
        """Return the canonical representative of the position-orbit with this rank."""
        total = self.count()
        if not (0 <= index < total):
            raise ValueError(f"index {index} outside range 0..{total - 1}")

        masks: list[Mask] = []
        occupied = 0
        subgroup = self.action.full_group

        for gi, need in enumerate(self.material):
            available = self.all_squares_mask & ~occupied
            for choice, new_subgroup in self._choice_rows(subgroup, available, need):
                block = self.count(gi + 1, occupied | choice, new_subgroup)
                if index >= block:
                    index -= block
                    continue
                masks.append(choice)
                occupied |= choice
                subgroup = new_subgroup
                break
            else:
                raise AssertionError("unrank failed to find a block")

        return tuple(masks)

    def squares_of(self, masks: tuple[Mask, ...]) -> list[list[int]]:
        return [bits(m) for m in masks]


def demo() -> None:
    action = chess_d4_action()

    # Example material: three anonymous piece types with multiplicities 1, 1, 2.
    # This could be WK, BK, two white knights, but the ranker treats all of them
    # merely as typed pieces occupying squares.
    ranker = SymmetryRanker(action, [1, 1, 2])
    n = ranker.precompute();
    print("number of symmetry-distinct placements:", n)

    position = (
        mask_from_squares([0]),      # type 0
        mask_from_squares([63]),     # type 1
        mask_from_squares([10, 20]), # type 2, two identical pieces
    )

    r = ranker.rank(position)
    q = ranker.unrank(r)
    print("rank:", r)
    print("canonical squares:", ranker.squares_of(q))
    assert q == ranker.canonicalize(position)
    assert ranker.rank(q) == r

    for i in range(10000):
        r = randrange(n)
        q = ranker.unrank(r)
        assert ranker.rank(q) == r
    print("OK")

if __name__ == "__main__":
    demo()
It does not precompute full tables (but it did speed things up when I asked for it).
I added the test of 10000 random positions. Apparently it works.
Rein Halbersma
Posts: 779
Joined: Tue May 22, 2007 11:13 am

Re: Tablebase suggestion

Post by Rein Halbersma »

syzygy wrote: Sat May 23, 2026 1:16 am So if you start by looking at square 63, and you let "empty" choose the range 0..binom(n-1,k)-1, then filling square 63 with the kth piece means you add binom(63,k) to the index. Same for square 62,61,...,0, and you recover the combinadics approach to ranking/unranking.
Actually, this is a coincidence. For the symmetry-reduced indexing, you need the cumulative sums over the tables. Happily, for binomials this is again a binomial thanks to the hockey-stick identity. https://en.wikipedia.org/wiki/Hockey-st ... ity#Proofs
So indeed you walk over the orbits of squares of the board. If the group of symmetries is the trivial group, you end up with the binomial case discussed above.
yes
Filling an orbit in the canonical way will usually take out some (and probably quite often all) of the remaining symmetries. I guess you could then determine the remaining orbits of the squares? If no symmetry is left, you end up with the easy binomial case. This is basically what I did in my manually coded ranking/unranking functions.

Of course there is no reduction in the remaining symmetries if an orbit stays empty. I guess this can be used to treat similar orbits (with same stabilizer) together, i.e. order them and consider the lowest-ordered orbit that receives a piece. (Perhaps your program does this.)
I guess I understand this, but I would have to think about how to then build the table and get everything right.
The 'n' in canom(n, k, 0b111) pins down the orbit being filled, right?
So the main trick is that you can forget about what happened before; only the value of k and the bitmask matter.
Yes, exactly as with binomials: 'n' here means: 'n' remaining orbits to fill with 'k' pieces, and 0b111 non-trivial symmetries left.
Actually, I changed it back to use stabilizer subgroups. The original formulation by ChatGPT used a pair of stabilizers: the currently active one and a target stabilizer. The resulting index then split the table into subsets with different final stabilizers. But that makes the overhead of the helper tables quite a bit bigger. It's important to keep the state small, otherwise the overhead quickly overwhelms the symmetry savings.
You can improve on 2^|G| by considering the subgroups of the symmetry group G instead, but your table builder will probably discover this by itself.
Yes, I have greatly reduced the helper tables in the mean time. First of all, you are right to use stabilizers rather than arbitrary subsets of symmetries. Although the program would only keep the actually used masks, not the entire universe of masks. But secondly, the cumulative tables also ran over all possible configurations of each orbit. With 8 squares and say 3 colors, that gives 3^8+1 entries. But if you only have 2 pieces, the actually number of configuration is 1 + 8 + 8 + 8 * 7 is much much smaller. So now I have sparsity in both the symmetry and the configuration space.
D_4 has 9 subgroups, so 10 possible groups of remaining symmetries. Each of them can occur as the stabilizer of a subset of an orbit.
Nine Men's Morris seems to have 19 possible groups of remaining symmetries.
I use the notation D_2n for the regular n-gon which has order 2n. So chess has D8 and draughts has its subgroup D4. For Nine Men's Morris the group is D8 x C2 for regular endgames (35 subgroups) and D8 x S3 for the special case of 3-3 endgames (120 subgroups). The 3-3 case is special because it has full 3-ring permutation symmetry, whereas normal endgames have inner-outer ring symmetry (but only for edge-edge and corner-corner squares, the orbit size are 8, 8, 4 and 4). The 19 subgroups corresponds to the D16 group, but that's not the symmetry group of NMM.
Maybe I will try to write a program that generates ranking/unranking functions for <= N pieces in a non-table style. Not to use in my generator but just to finally do what I would have liked to do automatically in 2005.
That should now entirely be possible.
syzygy
Posts: 5999
Joined: Tue Feb 28, 2012 11:56 pm

Re: Tablebase suggestion

Post by syzygy »

syzygy wrote: Sat May 23, 2026 8:25 pm It does not precompute full tables (but it did speed things up when I asked for it).
I added the test of 10000 random positions. Apparently it works.
It seemed quite fast on chess [1,1,2] (OK, quite trivial, but still).
It is very slow on 10x10 draughts [5,5].
After asking ChatGPT to speed it up for that case a couple of times, the precomputation step became very fast, but the calculation takes 12.7s for a single position.
The code generated from your (Rein's) post takes 0.08s on the same position.

Maybe I'll look into implementing it myself.