A different tablebase encoding format

[ZirconiumX]

DTM vs WDL

Consider the endgame KBNK WTM.

With a DTM tablebase, any probe at any depth gives an immediate, branch-ending result with an exact score, just as if the entire subtree had been searched up to 65 ply deep. That exact score can be backed up in the search and eventually be made visible to the user via mate announcement or a resignation.

With a WLD tablebase, a probe will only show "White wins" (or draws in a very few and obvious cases). Since nearly all of the score entries will be "White wins", it's no surprise that a high compression can be had. But a simple "White wins" score can't distinguish among wins of varying lengths, so it's necessary to have either some extra, specific code or a separate DTM tablebase. But why not use that separate DTM tablebase in the first place? If extra, specific code is needed, why not just have that code instead of any tablebase?

If tablebases are made, or at least made available, in uncompressed DTM format, then that means that a chess program author is free to use that data for building custom compressed versions or building WLD versions, or versions with both compression and WLD. But you can't go from WLD to DTM, and one compression scheme might be wholly inappropriate for differing endgames.

I have no interest with WLD. If someone wants to take my data to make a WLD tablebase, that doesn't bother me at all. The same if they want to use some compression scheme. But they're not forced into either; they have a choice and that's the point.

Fine, but a DTM tablebase without information as to who is winning is somewhat useless.

Sarcasm aside, a DTM tablebase probed during search is probably going to be either slow (probed too much, resulting in costs being greater than gains), or unhelpful (probed too little, resulting in costs being greater than gains). Getting the balance perfect is going to require code wizardry.

Or, you could, you know, probe WDL in search, which is faster, and leave finding the mate sequence to the search.

Matthew:out

[syzygy]

With a DTM tablebase, any probe at any depth gives an immediate, branch-ending result with an exact score, just as if the entire subtree had been searched up to 65 ply deep. That exact score can be backed up in the search and eventually be made visible to the user via mate announcement or a resignation.

With a WLD tablebase, a probe will only show "White wins" (or draws in a very few and obvious cases). Since nearly all of the score entries will be "White wins", it's no surprise that a high compression can be had. But a simple "White wins" score can't distinguish among wins of varying lengths, so it's necessary to have either some extra, specific code or a separate DTM tablebase. But why not use that separate DTM tablebase in the first place? If extra, specific code is needed, why not just have that code instead of any tablebase?

First, you need to realise, if you don't do this already, that the extra information is ONLY necessary once the game has reached a tablebase position. Within the search tree it is sufficient to probe WDL data.

Probing only WDL during the search has the huge advantage over probing DTM that the total amount of data being probed is much smaller. Therefore the same amount of available RAM will have a much bigger impact on probing efficiency.

In addition, it does not cost anything to keep separate WDL tables, as this information can then be removed from the DTM or DTZ tables. Those will compress to smaller files.

(A further advantage is that only probing DTM/DTZ allows you to throw away the "bigger half" (wtm/btm) of each table, further reducing the size of DTM/DTZ by about two thirds.)

If tablebases are made, or at least made available, in uncompressed DTM format, then that means that a chess program author is free to use that data for building custom compressed versions or building WLD versions, or versions with both compression and WLD. But you can't go from WLD to DTM, and one compression scheme might be wholly inappropriate for differing endgames.

The uncompressed data is not suitable for custom compression schemes that take into account, for example, that a particular position can be won by a capture. The generator and compressor are not fully independent.

I have no interest with WLD. If someone wants to take my data to make a WLD tablebase, that doesn't bother me at all. The same if they want to use some compression scheme. But they're not forced into either; they have a choice and that's the point.

I predict very little interest in uncompressed 6-piece tables that are impossible to download, even today still too big to store, and that only represent may be 20% of the actual work involved in producing a practical TB solution.