Tablebase suggestion

[Geezer one]

I was thinking that partial tablebases could be useful. My idea is to have only 1 minor/major piece, plus an extra queen, allowed per side. This will cut down on size a lot. The majority of endgames under 10 pieces fall into this category. For the engines, I thought that a simple check if the game is in this condition will turn on the tablebase.

[jkominek]

I was thinking that partial tablebases could be useful. My idea is to have only 1 minor/major piece, plus an extra queen, allowed per side. This will cut down on size a lot. The majority of endgames under 10 pieces fall into this category. For the engines, I thought that a simple check if the game is in this condition will turn on the tablebase.

I may be revealing myself as the true Geezer here, because I have heard this suggestion from well meaning tablebase fans for thirty years. On the surface it's not a bad notion. But what ideas do you have for efficiently generating partial tablebase subsets near the top of the mountain? That's the gotcha that has myself and others stumped.

[Geezer one]

Subsets near the top of the mountain? I don't understand what that means. I thought a tablebase maker could simply omit a lot of the options when creating one, by removing combinations of pieces.

[syzygy]

I was thinking that partial tablebases could be useful. My idea is to have only 1 minor/major piece, plus an extra queen, allowed per side. This will cut down on size a lot. The majority of endgames under 10 pieces fall into this category. For the engines, I thought that a simple check if the game is in this condition will turn on the tablebase.

Tablebase files are usually per piece combination (and sometimes per side-to-move), so this cuts down on the number of files, not on the size of each file.

[syzygy]

Subsets near the top of the mountain? I don't understand what that means. I thought a tablebase maker could simply omit a lot of the options when creating one, by removing combinations of pieces.

To correctly generate a tablebase with a pawn, you need to have previously generated the tablebases with the pawn promoted to a queen, rook, bishop, and knight. (Or you could limit to queen promotions and accept that you get incorrect results. You could then try to keep track of the assumptions you made and use those to correct the results returned by TB probes, and it all becomes an impractical mess that nobody will ever implement.)

[jkominek]

Subsets near the top of the mountain? I don't understand what that means. I thought a tablebase maker could simply omit a lot of the options when creating one, by removing combinations of pieces.

Every table has a dependency tree. That is to say, building a table for a specific combination of pieces requires having previously built (and kept available) all tables that it can convert into through capture or promotion. This dependency is recursive. With 9 men endgames the "top of the mountain" is king and 4 pawns versus king and 3 pawns.

Even pawnless tables have complicated dependencies. In 2013, on April 1 to be exact, the creators of the 7-man DTM Lomonosov tablebases made an internet newsgroup posting claiming to have generated the 9-man table KRRBN versus KQBN, one of the more complicated of pawnless 9-man combinations. It was purportedly accomplished using a quantum computer located at the Laboratory of Quantum Informatics of Moscow State University, with money provided by several unnamed Arab sheiks. In case the clue is not obvious, it was an April Fools joke.

Behold the dependency tree.

[syzygy]

Subsets near the top of the mountain? I don't understand what that means. I thought a tablebase maker could simply omit a lot of the options when creating one, by removing combinations of pieces.

Every table has a dependency tree. That is to say, building a table for a specific combination of pieces requires having previously built (and kept available) all tables that it can convert into through capture or promotion. This dependency is recursive. With 9 men endgames the "top of the mountain" is king and 4 pawns versus king and 3 pawns.

Or a little more precise: for 5v4 men endgames. The 6v3, 7v2 and 8v1 endgames are independent.

What one can also do is calculate 8- or 9-men tablebases only for positions with at least one pair of opposing pawns. An 8-men position with opposing pawns is only dependent on 7-men positions and other 8-men positions with opposing pawns, and the full 7-men set is already available. For 9-men positions with 1 opposing pair you would still need the full 8-men set, though (or at least the applicable ones from 4v4, 5v3, 6v2, 7v1). For 9-men positions with at least 2 opposing pairs you would only need 8-men positions with at least 1 opposing pair (unless I am overlooking something, I did not try to think this through completely).

The 8-men positions with 1 opposing pair have been computed by Marc Bourzutschky (DTC metric) and are accessible here:
https://op1-tables.info/

8-men with 1 opposing pair only needs 82.5 GB of RAM with 1 byte per position if you split it up in pawn slices with the pair fixed, so it does not need a big machine.
9-men with 2 opposing pairs should only need 1.2 GB of RAM with 1 byte per position.
Why not go further and do 10-men with 3 opposing pairs, 11-men with 4 opposing pairs and 12-men with 5 opposing pairs (and 2 kings).

The 32-men starting position has 8 opposing pairs, so you only need to compute the full 24-men set first

Even pawnless tables have complicated dependencies. In 2013, on April 1 to be exact, the creators of the 7-man DTM Lomonosov tablebases made an internet newsgroup posting claiming to have generated the 9-man table KRRBN versus KQBN, one of the more complicated of pawnless 9-man combinations. It was purportedly accomplished using a quantum computer located at the Laboratory of Quantum Informatics of Moscow State University, with money provided by several unnamed Arab sheiks. In case the clue is not obvious, it was an April Fools joke.

By chance that was also the exact date that I released my (then 6-men) tablebase generator.

[syzygy]

And 7-men with 1 opposing pair of pawns (and perhaps 8-men with 2, etc.) would probably be very useful for regular users.

[jkominek]

Or a little more precise: for 5v4 men endgames. The 6v3, 7v2 and 8v1 endgames are independent.

What one can also do is calculate 8- or 9-men tablebases only for positions with at least one pair of opposing pawns. An 8-men position with opposing pawns is only dependent on 7-men positions and other 8-men positions with opposing pawns, and the full 7-men set is already available. For 9-men positions with 1 opposing pair you would still need the full 8-men set, though (or at least the applicable ones from 4v4, 5v3, 6v2, 7v1). For 9-men positions with at least 2 opposing pairs you would only need 8-men positions with at least 1 opposing pair (unless I am overlooking something, I did not try to think this through completely).

The 8-men positions with 1 opposing pair have been computed by Marc Bourzutschky (DTC metric) and are accessible here:
https://op1-tables.info/

8-men with 1 opposing pair only needs 82.5 GB of RAM with 1 byte per position if you split it up in pawn slices with the pair fixed, so it does not need a big machine.
9-men with 2 opposing pairs should only need 1.2 GB of RAM with 1 byte per position.
Why not go further and do 10-men with 3 opposing pairs, 11-men with 4 opposing pairs and 12-men with 5 opposing pairs (and 2 kings).

Yes you're correct. It may have been you, or from hgm, who introduced me to the idea of treating pawn slices as independently computable sub-tables.

I've made similar RAM estimates myself for blocked pawn sub-table construction. I'll admit: there is a certain satisfaction to (somehow) acquiring a Top-10 supercomputer to complete all of 8-men, then all of 9-men, then all of 10-men, etc. and donating them to the world. However a better ambition may be irregular construction that maximizes coverage of endgame positions seen in computer play. The first appearance of some 9-men tables containing a pair or two of opposing pawns should garner considerable interest among grandmasters and endgame specialists.

[Ajedrecista]

Hello Ronald:

[...]

The 8-men positions with 1 opposing pair have been computed by Marc Bourzutschky (DTC metric) and are accessible here:
https://op1-tables.info/

8-men with 1 opposing pair only needs 82.5 GB of RAM with 1 byte per position if you split it up in pawn slices with the pair fixed, so it does not need a big machine.
9-men with 2 opposing pairs should only need 1.2 GB of RAM with 1 byte per position.
[...]

I knew about the existance of some 8-man EGTB, but I did not know about their public availability! Thank you very much for pointing to them.

Regards from Spain.

Ajedrecista.