Tablebase suggestion

[syzygy]

Here is another exciting thing to consider. If we can get tablebases up to 9 or 10 pieces (with Q promotion only, limited minor/major pieces, no extreme material imbalances etc.) we can 'half' solve chess.

But ignoring underpromtions is not going to make the difference. It won't help you get anywhere near 9 let alone 10 pieces. And "solve" should not be used in the same sentence as "tablebases with Q promotion only"...

Anyway, feel free to start implementing them.

[Geezer one]

But ignoring underpromtions is not going to make the difference. It won't help you get anywhere near 9 let alone 10 pieces. And "solve" should not be used in the same sentence as "tablebases with Q promotion only"...

Anyway, feel free to start implementing them.

If Q promotions only was combined with other conditions, such as no more than 1 minor/major piece each + additional Queen, and no extreme imbalance, would that not make a big difference? Roughly it should be at least 10x less positions, possibly much more.

[hgm]

You will need to consider the nonsensical ones to be sure that you get the sensical ones right.

Not really. If you assume conversion to the non-sensical one is lost for the strong side by default (i.e. without 'considering it'), and a predecessor still is totallly won because there are sufficient conversions to sensical successors, you will have the sensical predecessor right. This is basically the definition of non-sensical: P-slices the strong side would never have to visit to secure the win.

There could be some discussion about what 'getting it right' means, though. By declaring some conversions a loss by default, while they actually could be a win, some winning positions might prove their win by forcing another winning conversion, taking more moves to force it. So their DTZ would not be the same as when you would have solved the non-sensical P-slice to get correct WDL values for all conversions to it. (This only applies to P-slices that use the non-sensical one as direct successor: they might have deviating DTZ, but if they have the correct WDL (e.g. because the DTZ is never pushed above 100 ply) their predecessors are fully identical to those for a complete DTZ solution.)

But so what? DTZ is not sacred. Playing according to DTZ in general gives idiotic play, first shedding all unnecessary material until a lengthy win with an absolute minimum of advantage is reached. Going for the fastest conversion is in general not the way to get the fastest win. Avoiding conversions to unclear P-slices is likely to do more good than harm in this respect. DTNSZ ('distance to non-sacrificial zeroing') or DTNNZ ('distance to no-nonsense zeroing') could be much more desirable.

We have gone through this before. There will always be positions where you need an underpromotion and many more positions where a promotion is not winning. "Usually" is not enough to "correctly generate a tablebase".

This again seems to argue from the POV that a 'tablebase' is a monolithic thing containing all P-slices, and that it is not 'correctly generated' untill all these P-slices have been solved. While in practice the P-slices are largely independent, and even having a small fraction of the P-slices for a given material composition can be very useful. As long you have the P-slices that are most likely to occur in actual games. Quite similar to a sitiation where you don't have the full 7-man set, but only KRPPKRP; having KQQQKQQ (or even KQQRKQR) in addition will be zero help for winning more games.

You can know which P-slices might be tainted by ignoring underpromotion, and simply leave those out. If you have solved a P-slice under assumption that all conversions to a certain successor are losses, you can then check whether the positions from which such conversions are possible still remain wins. If they are the result of the conversion was immaterial, basically subject to alpha-beta cutoff.

If there are any positions the WDL value is dependent on the value of 'forbidden' conversions, this can be treated in several ways. You could simply leave out the P-slice from the set. You can also build the P-slice in WDLU format, introducing a new result Undecided. Conversions into the P-slice that probe a U position could by default assume a loss for the strong side, repeating the procedure during retrograde propagation through the P-slices. If the number of such conversions is very small, you can try to repair it by investigating these conversions through alpha-beta search, possibly considering underpromotion in that search. Usually the problem is a very quick stalemate, that promotion to Rook would cure.

[Ender8pieces]

IMHO, I think approach as geezer one and hgm could give practical value, maybe tens point of elo for engine. It won't be theoretical accurate but we can bound the probability for failure.
I know syzygy has a lot of experience and knowledge but I wonder if we can't find some balance. We might not call it tablebases but it should be much more rigorous than usual engines.

I also realize this approach might be "unpopular" for those who cherish the mathematical elegance of optimal play and exhaustive discovery. However, it might be better shifting the goal from theoretical perfection to practical engine utility.

Summarize the approach and some thoughts of mine, we can have the next principles:
Prioritize Realistic Endgames: Focus primarily on Rook endgames and their common branches, as they offer the highest return on investment for competitive play.

Simplified Search Space: Ignore underpromotions unless they are tactically essential (I would be happy to see counter example, underpromotion with compensation after more than 3 moves).

Cap the Depth (DTZ < 100): Align with FIDE’s practical rules. Focusing on wins within 50 moves covers the vast majority of human and engine competitive needs.

Probabilistic Accuracy (Error Tolerance): Accept a very low, pre-calculated probability of error in exchange for massive efficiency gains. While most data will be deterministic, a small fraction may be estimated. This "engineering approach" allows us to gradually expand the "certainty zones" in the most critical parts of the table.

Exclude Highly Asymmetric Material: Avoid calculating configurations like
KQQQkrrr. Modern engines already handle these positions with high proficiency, and except for extreme edge cases, the win for the stronger side is practically guaranteed.

Proposed POC (Proof of Concept):
Although pawnless endgames are more complex to solve, they are the necessary foundation before tackling pawn-based endings.
Ideally, the goal is to solve KQQRkqqr, as it represents the main branch of the highly common Rook + 2 Pawns endings. Practically, For the initial proof of concept, I am focusing on KQQQkqqq. This will serve as a baseline to test memory management and the error-tolerance model.

Resource Constraints: I am developing this using relatively modest hardware: 1TB of storage and 64GB of RAM.

I assume some of these ideas have been discussed here in the past. I would be very interested to hear your thoughts, advice, and comments, especially if anyone has attempted similar directions before.

[hgm]

The point of my argument is that any knowledge on KQQRkqqr is completely unnecessary for calculating the result of any KRPPkrpp position we will encounter in practice. Does there exist any GM game where KRPPkrpp was won through a position that has 4 Queens on the board? Normal course of KRPPkrpp endings is that the winning site converts to KQRPkrpp, which then immediately converts to KRPkpp by trading R for Q on the promotion square. If capturing the Queen on the promotion square is not possible, conversion to KRPKpp usually takes place directly from KRPPkrpp, as the opponent will be happy to sac his Rook to avoid having to face an extra Queen. If none of that is possible, the advantage in KQRPkrpp is so large that you never have to allow a Queen for the opponent, and can easily win without needing a second Queen yourself.

[Ender8pieces]

hgm, thank you for your insight.

If I understand your point correctly, you are suggesting that we can bypass the complexity of KRPPkqqr by starting our analysis from KRPPkrp where the Rook has just captured a newly promoted Queen. Since this 7-piece endings are already fully solved, working backwards from there to KRPPkrpp is a significantly lighter task than solving the 4-Queen transition.

However, I have a concern regarding the exclusion of mutual promotions.

While it is true that a '2-Queen battle' is rare in grandmaster games, I suspect it plays a much more significant role in the engine's search tree. An engine might evaluate a line as a draw or a loss simply because it cannot see the outcome of a mutual promotion race, whereas in reality, one side might have a forced win in the resulting Queen ending.

In my view, while underpromotion is an extreme rarity that can be safely neglected, mutual promotion to Queens happens often enough in deep tactical tactical lines to warrant inclusion.

That said, I don’t dismiss your approach. It would be a fascinating experiment to build a tablebase based on your 'simplified conversion' theory and test it against a standard engine on KRPPkrpp states. If my hypothesis is correct, a standard engine would exploit the 'blind spot' of mutual promotions to outplay the simplified version.

It's essentially an empirical question: how much 'truth' can we slice away before the engine's strength begins to suffer?

[Ender8pieces]

"hgm, thank you for your insight.

If I understand your point correctly, you are suggesting that we can bypass the complexity of KRPPkqqr....."

Sorry, meant KRPPkrpp

[syzygy]

You will need to consider the nonsensical ones to be sure that you get the sensical ones right.

Not really. If you assume conversion to the non-sensical one is lost for the strong side by default (i.e. without 'considering it'), and a predecessor still is totallly won because there are sufficient conversions to sensical successors, you will have the sensical predecessor right. This is basically the definition of non-sensical: P-slices the strong side would never have to visit to secure the win.

You just need one position where there is no winning conversion to sensical successors. Then you need to check the nonsensical successor to be sure. In some cases you might perhaps be able to calculate the result using a search instead of generating the nonsensical table, but that could get slow things down terribly and probably will often not work.

But so what? DTZ is not sacred. Playing according to DTZ in general gives idiotic play, first shedding all unnecessary material until a lengthy win with an absolute minimum of advantage is reached.

As long as you don't get close to the 50-move rule this will indeed not matter for game play.

We have gone through this before. There will always be positions where you need an underpromotion and many more positions where a promotion is not winning. "Usually" is not enough to "correctly generate a tablebase".

This again seems to argue from the POV that a 'tablebase' is a monolithic thing containing all P-slices, and that it is not 'correctly generated' untill all these P-slices have been solved.

I responded to what you wrote and in the order you wrote it. You are now rearranging stuff, taking my statement out of context. You were arguing that what I had written in response to "Geezer one" was not right. (So yeah, apparently you then changed the entire premise from tables to P-slices. Duh..... tiresome. So I guess you do agree with what I responded to Geezer one, you just somehow wanted to confuse things.)

I believe in the past and perhaps also now you have proposed two different things:
1. Make assumptions like "underpromotions are not necessary", "two queens are not necessary". This results in incorrect tablebases, but one can of course decide to accept rare mistakes.
2. Generate only "sensical" tables but only for those pawn slices which are "provably" correct. I suspect this is not going to be a very effective method basically for the reason that there will always be positions in any pawn slice of a sensical table whose WDL value, somewhere down the line, relies on the WDL value of some position in a nonsensical table. Since you did not generate the latter, you cannot generate the former.

[syzygy]

I know syzygy has a lot of experience and knowledge but I wonder if we can't find some balance.

I am not stopping anyone from implementing their ideas.

[Ender8pieces]

1. Make assumptions like "underpromotions are not necessary", "two queens are not necessary". This results in incorrect tablebases, but one can of course decide to accept rare mistakes.
2. Generate only "sensical" tables but only for those pawn slices which are "provably" correct. I suspect this is not going to be a very effective method basically for the reason that there will always be positions in any pawn slice of a sensical table whose WDL value, somewhere down the line, relies on the WDL value of some position in a nonsensical table. Since you did not generate the latter, you cannot generate the former.

I'd like to better understand your perspective on the first approach you mentioned (making assumptions like ignoring underpromotions).

I am curious if your objection is primarily rooted in the pursuit of mathematical perfection and the theoretical integrity of tablebases, or if you believe that such 'imperfect' tables would simply fail to provide any practical Elo gain for current engines?

In your experience, how significant is the impact of underpromotions in these types of endings (KRPPkrpp)? If we accept a certain margin of error in these rare cases to make the 8-piece slices computationally feasible on consumer hardware, do you suspect the resulting 'noise' would negate the benefits, or is there a pragmatic path here for engine improvement?

Would love to hear any insights or advice you might have on the practical side of this compromise.