Tablebase suggestion

[hgm]

I believe in the past and perhaps also now you have proposed two different things:
1. Make assumptions like "underpromotions are not necessary", "two queens are not necessary". This results in incorrect tablebases, but one can of course decide to accept rare mistakes.
2. Generate only "sensical" tables but only for those pawn slices which are "provably" correct. I suspect this is not going to be a very effective method basically for the reason that there will always be positions in any pawn slice of a sensical table whose WDL value, somewhere down the line, relies on the WDL value of some position in a nonsensical table. Since you did not generate the latter, you cannot generate the former.

Sorry if changing the order of my quotes misrepresented your argument; I added the second reaction as an afterthought after addressing what I considered the most important poiint I wanted to address.

Now and in the past I have always argued from the perspective of P-slices; from a theoretical point of view what you call 'tablebases' (presumably meaning all positions with a given material composition) has no special significance; it is just a collection of subsets of positions connected by reversible moves (the P-slices, which need to be solved all at once) connected by irreversible moves to define an order relation between them that allows sequential generation. Just like some material compositions will occur more frequently than others (e.g. KRPPkrpp a lot, KQQQkqqq never). So as far as I am concerned the relevant thing is to generate P-slices that are useful in practice, and that KRPPkrpp also has a small fraction of P-slices that virtually never occur, and cannot be solved should not discourage one from solving those that you can.

I do not propose to acceptany mistakes. You can easily detect which P-slices are incorrect. Assumptions about what is not necessary are used to ignore conversions to them during the building, which is equivalent to assuming converting to them is a loss for the strong side. You can then solve again under the assumption that is a win for that side, to see if any of the positions in the P-slice would change its WDL. If not, the P-slice was solved fully correct WDL-wise, and can be used as successor in a DTZ build of P-slices convering to it without even affecting the DTZ values in them. If some positions change WDL you can simply add the P-slice to the set that you could not do.

Note that most P-slices have no promotions at all, because there are no Pawns on 7th rank. In fact most P-slices have the most advanced Pawns quite far from 8th rank, and retrograde solving must pass through many P-slices closer to promotion. Each such step tend to cleanse positions affected by the promotion assumptions from the EGT, because the retrograde propagation tends to find alternative winning conversions, so that they get the same WDL even though they have different DTZ, and DTZ errors do not propagate through predecessors if the WDL is correct. So I think you are way to pessimistic when assuming reasonable promotion assumptions (whether they are about underpromotion or the number of allowed surviving promotion pieces) would spoil the majority of P-slices.

E.g. in the position
[d]8/5P1k/5K2/8/8/8/8/8 w
1. f8=Q would spoil the win by checkmating, and would not have DTZ=1 ply that promotion to Rook would give it. But since white can play 1. Ke7 the position is still easily won, except that it now gets DTZ=3 ply. If the white King would have been at h5 it is more serious: 1. f8=R would be a win, but any other move will draw because 1... Kg7 will lose you the Pawn. But what was black's previous move here? (Which the generation will now try to make retrogradely.) The only case where he was not already next to Pawn or King (and then would avoid losing by capture that) is h8, and from there Kg7 would have drawn even with when considering underpromotion. So the damage remains limited to a single U position in the entire P-slice.

This makes it impossible for the P-slice with Pf6 to use 1. f6-f7 as a winning conversion, but it can play 1. Kg5 instead (1... Kg8 2. Kg6 Kf8 3. f7 or 2... Kh8 3. Kf7 Kh7 4. Ke7) to win that position in an alternative way, not suffering any WDL damage.

[syzygy]

Note that most P-slices have no promotions at all, because there are no Pawns on 7th rank. In fact most P-slices have the most advanced Pawns quite far from 8th rank, and retrograde solving must pass through many P-slices closer to promotion. Each such step tend to cleanse positions affected by the promotion assumptions from the EGT, because the retrograde propagation tends to find alternative winning conversions, so that they get the same WDL even though they have different DTZ, and DTZ errors do not propagate through predecessors if the WDL is correct. So I think you are way to pessimistic when assuming reasonable promotion assumptions (whether they are about underpromotion or the number of allowed surviving promotion pieces) would spoil the majority of P-slices.

Yes, perhaps I am too pessimistic here. My experience is that everything in a tablebase which can go wrong will go wrong, i.e. any assumption will be refuted by some counterexample. But I get the point that there are probably hardly any winning or drawing positions that crucially depend on an underpromotion many (pawn) moves down the line. So this could be worth a try, although it will be a major headache to implement.

[syzygy]

1. Make assumptions like "underpromotions are not necessary", "two queens are not necessary". This results in incorrect tablebases, but one can of course decide to accept rare mistakes.
2. Generate only "sensical" tables but only for those pawn slices which are "provably" correct. I suspect this is not going to be a very effective method basically for the reason that there will always be positions in any pawn slice of a sensical table whose WDL value, somewhere down the line, relies on the WDL value of some position in a nonsensical table. Since you did not generate the latter, you cannot generate the former.

I'd like to better understand your perspective on the first approach you mentioned (making assumptions like ignoring underpromotions).

I am curious if your objection is primarily rooted in the pursuit of mathematical perfection and the theoretical integrity of tablebases, or if you believe that such 'imperfect' tables would simply fail to provide any practical Elo gain for current engines?

I don't understand how you read an objection in my first observation.

[Ender8pieces]

Now and in the past I have always argued from the perspective of P-slices; from a theoretical point of view what you call 'tablebases' (presumably meaning all positions with a given material composition) has no special significance; it is just a collection of subsets of positions connected by reversible moves (the P-slices, which need to be solved all at once) connected by irreversible moves to define an order relation between them that allows sequential generation. Just like some material compositions will occur more frequently than others (e.g. KRPPkrpp a lot, KQQQkqqq never). So as far as I am concerned the relevant thing is to generate P-slices that are useful in practice, and that KRPPkrpp also has a small fraction of P-slices that virtually never occur, and cannot be solved should not discourage one from solving those that you can.

I do not propose to acceptany mistakes. You can easily detect which P-slices are incorrect. Assumptions about what is not necessary are used to ignore conversions to them during the building, which is equivalent to assuming converting to them is a loss for the strong side. You can then solve again under the assumption that is a win for that side, to see if any of the positions in the P-slice would change its WDL. If not, the P-slice was solved fully correct WDL-wise, and can be used as successor in a DTZ build of P-slices convering to it without even affecting the DTZ values in them. If some positions change WDL you can simply add the P-slice to the set that you could not do.

Note that most P-slices have no promotions at all, because there are no Pawns on 7th rank. In fact most P-slices have the most advanced Pawns quite far from 8th rank, and retrograde solving must pass through many P-slices closer to promotion. Each such step tend to cleanse positions affected by the promotion assumptions from the EGT, because the retrograde propagation tends to find alternative winning conversions, so that they get the same WDL even though they have different DTZ, and DTZ errors do not propagate through predecessors if the WDL is correct. So I think you are way to pessimistic when assuming reasonable promotion assumptions (whether they are about underpromotion or the number of allowed surviving promotion pieces) would spoil the majority of P-slices.

I’d like to better understand your approach to P-slices, specifically regarding the base cases.As I understand, retrograde solving starts from the chronologically 'last' slices (pawns on the 7th rank) and moves backward. However, these slices eventually convert into 8-piece no-pawn positions (like KQQRkqqr), which are arguably the hardest to solve and require massive RAM.
If we don't have the full 8-piece no-pawn tablebases as a reference, how does your verification mechanism function? Are you suggesting ,as discussed before,we assume the pawn is immediately captured after promotion (reducing it to a known 7-piece ending), or are you describing a way to verify the slice's WDL without having the successor no-pawn table at all?
Using an engine as a heuristic for the base case is one option, but you seem to be describing a more formal verification process. I’d love to hear how you handle that 'missing link' of the no-pawn 8-piece endings.

[hgm]

It is a sort of general principle in chess that a given advantage carries more weight when there is in total less material on the board. Based on that, it seems poor strategy for a would-be winner to pump up the number of Queens on both sides. So we put some restrictions on this would-be winner: he is not allowed to grant the opponet a surviving promotion, and he is not allowed to have more than one piece obtained through promotion on the board at any time. (If the opponent will never have any such piece, having one extra Queen should be enough to force a win.)

That means the number of reachable successor end-games will remain limited; from KRPPPkrppp you can reach KQRPPkrppp, but no 10-men with fewer Pawns. You can only reach subsets of those two endings. If a 10-men position is won under these restrictions, it would certainly be won without them. If a position is not won under these restrictions, it might be winnable without them. But this is quite unlikely. Because it is unlikely that allowing your opponent an extra Queen would improve your chances, or that without doing that the advantage of a full Queen would not be sufficient to win. So in general you would find most forced wins this way.

In practice the restrictions are implemented by assuming the forbidden promotions result in a loss. While opponent promotions are treated by allowing his Pawns to enter 1st rank, where they would represent an unspecified promotion piece, and declare a win for a side to move that has a Pawn there. This gives the would-be winner the opportunity to capture the promotion piece in his own turn to avoid the loss, and thus directly convert to the EGT where the opponent has one fewer Pawn, but no extra pieces. (Which then might convert to the EGT where the would-be winner lacks the capturer, when the promotion square was protected. But if the piece thus sacrificed was itself obtained by promotion, this then opens the possibility to promote one of his other Pawns.)

How realistic the assumptions are can be tested by making the reverse assumption, e.g. that a second promotion while the piece from teh first is still on the board automatically results in a win. By comparing the two EGT that result from these opposit assumptions, you can see which non-won positions have turned into wins. This should be a small fraction of the positions only, and they could be recorded in the EGT as 'undecided'. This would not cause EGT probing to ever produce a false result; in some cases it would just cause no result. This would still be helpful to an engine; when it is not happy with the results of the positions that have one, its search would be guided along the undecided lines to the forbidden conversion without much branching, and it could then continue a normal alpha-beta search in the end-game we never generated the EGT for.

[Ender8pieces]

It is a sort of general principle in chess that a given advantage carries more weight when there is in total less material on the board. Based on that, it seems poor strategy for a would-be winner to pump up the number of Queens on both sides. So we put some restrictions on this would-be winner: he is not allowed to grant the opponet a surviving promotion, and he is not allowed to have more than one piece obtained through promotion on the board at any time. (If the opponent will never have any such piece, having one extra Queen should be enough to force a win.)

That means the number of reachable successor end-games will remain limited; from KRPPPkrppp you can reach KQRPPkrppp, but no 10-men with fewer Pawns. You can only reach subsets of those two endings. If a 10-men position is won under these restrictions, it would certainly be won without them. If a position is not won under these restrictions, it might be winnable without them. But this is quite unlikely. Because it is unlikely that allowing your opponent an extra Queen would improve your chances, or that without doing that the advantage of a full Queen would not be sufficient to win. So in general you would find most forced wins this way.

In practice the restrictions are implemented by assuming the forbidden promotions result in a loss. While opponent promotions are treated by allowing his Pawns to enter 1st rank, where they would represent an unspecified promotion piece, and declare a win for a side to move that has a Pawn there. This gives the would-be winner the opportunity to capture the promotion piece in his own turn to avoid the loss, and thus directly convert to the EGT where the opponent has one fewer Pawn, but no extra pieces. (Which then might convert to the EGT where the would-be winner lacks the capturer, when the promotion square was protected. But if the piece thus sacrificed was itself obtained by promotion, this then opens the possibility to promote one of his other Pawns.)

How realistic the assumptions are can be tested by making the reverse assumption, e.g. that a second promotion while the piece from teh first is still on the board automatically results in a win. By comparing the two EGT that result from these opposit assumptions, you can see which non-won positions have turned into wins. This should be a small fraction of the positions only, and they could be recorded in the EGT as 'undecided'. This would not cause EGT probing to ever produce a false result; in some cases it would just cause no result. This would still be helpful to an engine; when it is not happy with the results of the positions that have one, its search would be guided along the undecided lines to the forbidden conversion without much branching, and it could then continue a normal alpha-beta search in the end-game we never generated the EGT for.

I understand the logic of using bounds: if a position yields the same result under both 'Assume Loss' and 'Assume Win' for the missing conversion, its value is indeed mathematically proven regardless of the true value of that missing link.

However, my main concern is empirical rather than theoretical. By treating the missing tablebases as a variable bound, we might end up with a tablebase where a very large fraction—perhaps the majority—of positions are flagged as 'Undecided'.

[Ender8pieces]

1. Make assumptions like "underpromotions are not necessary", "two queens are not necessary". This results in incorrect tablebases, but one can of course decide to accept rare mistakes.
2. Generate only "sensical" tables but only for those pawn slices which are "provably" correct. I suspect this is not going to be a very effective method basically for the reason that there will always be positions in any pawn slice of a sensical table whose WDL value, somewhere down the line, relies on the WDL value of some position in a nonsensical table. Since you did not generate the latter, you cannot generate the former.

I'd like to better understand your perspective on the first approach you mentioned (making assumptions like ignoring underpromotions).

I am curious if your objection is primarily rooted in the pursuit of mathematical perfection and the theoretical integrity of tablebases, or if you believe that such 'imperfect' tables would simply fail to provide any practical Elo gain for current engines?

I don't understand how you read an objection in my first observation.

Fair point. I certainly accept the skepticism—as you've explained in other posts, EGT generation is a delicate business where errors can propagate easily.

That said, I view the removal of underpromotions as a necessary pragmatic trade-off. While it is a compromise on theoretical perfection, it offers a massive reduction in both storage and required compute power.

My hope is that by focusing on the most common, near-balanced material compositions, the utility gained for engine analysis will outweigh the loss of edge-case accuracy

[syzygy]

I’d like to better understand your approach to P-slices, specifically regarding the base cases.As I understand, retrograde solving starts from the chronologically 'last' slices (pawns on the 7th rank) and moves backward. However, these slices eventually convert into 8-piece no-pawn positions (like KQQRkqqr), which are arguably the hardest to solve and require massive RAM.
If we don't have the full 8-piece no-pawn tablebases as a reference, how does your verification mechanism function? Are you suggesting ,as discussed before,we assume the pawn is immediately captured after promotion (reducing it to a known 7-piece ending), or are you describing a way to verify the slice's WDL without having the successor no-pawn table at all?
Using an engine as a heuristic for the base case is one option, but you seem to be describing a more formal verification process. I’d love to hear how you handle that 'missing link' of the no-pawn 8-piece endings.

My explanation:
First you decide which set of tablebases (piece/pawn combinations) are sensical.
Then you generate them P-slice by P-slice in the right order. It will turn out that some P-slices cannot be generated, and those you will simply skip. (You'll have to mark them somehow in your tablebase data.)
Each P-slice is generated twice:
- during the first generation pass, you set all successor positions in non-sensical tables to win-by-white.
- during the second generation pass, you set the same successor positions in non-sensical tables to win-by-black.
If the two generations result in the exact same WDL data for the P-slice, then you know that the generated WDL data for the whole slice is accurate. In that case you keep the P-slice, otherwise you throw it away and skip the P-slice.

If the final result is disappointing, you can broaden your initial definition of "sensical" tablebases and start over.

So you can limit the "sensical tables" to tables with a limited number of pieces. These tables can all be generated easily on regular hardware, because each P-slice generation is an independent computation.

It could be very interesting to implement this approach not necessarily for generating tablebases for use during a search but to solve a given endgame position. Then you don't need to (try to) generate and store all P-slices but just those that are relevant to the position. I think a similar approach, perhaps even the same approach, has been implemented in the past in programs like Freezer and Finalgen, but as far as I am aware these are now older programs and can probably be improved upon.

Using the approach for "regular" tablebases requires making a lot of somewhat arbitrary choices regarding which tables are "sensical" and which are not. I would rather just go for the complete files (but tablebases with one or more paris of opposing pawns are very interesting).

[Geezer one]

Using the approach for "regular" tablebases requires making a lot of somewhat arbitrary choices regarding which tables are "sensical" and which are not. I would rather just go for the complete files (but tablebases with one or more paris of opposing pawns are very interesting).

There was a very interesting suggestion I read, which was using an engine to selectively get rid of positions that were too far from a drawn eval.

[syzygy]

Using the approach for "regular" tablebases requires making a lot of somewhat arbitrary choices regarding which tables are "sensical" and which are not. I would rather just go for the complete files (but tablebases with one or more paris of opposing pawns are very interesting).

There was a very interesting suggestion I read, which was using an engine to selectively get rid of positions that were too far from a drawn eval.

I don't know how to understand that as a meaningful suggestion to tablebase generation.

If you mean for compression, then it is basically useless. You can use a regular search to predict W/D/L and use this to improve compression, but it takes way too long to do this for all positions individually, and you would need to use the exact same engine for probing positions (which again will be quite slow). Wasn't this already discussed and dismissed earlier in this thread?

If you fix the side to move, many/most tablebases will be generally won for white or black or generally drawn, so you could use this as a prediction for all positions, but tablebase compression already takes care of this.