Isaac wrote:1)Do you think there is still room for improvement in terms of compression of the Syzygy as they are now? I mean using conventional programming and not quantum computers programming if that makes a difference.
If you let the probing code do more, you can get the TBs smaller. It is a space-time tradeoff.
For example, I believe Shredderbases do not store the correct WDL value for in-check positions. The probing code therefore has to do extra work when probing such positions (basically try all check-evasion moves, probe the resulting positions, and back up the result). Instead of the correct WDL value, the TB can now store a value that compresses better.
My TBs don't use this trick for in-check positions, but it does use the trick for positions with winning and drawing captures.
If there is a winning capture, the position is won and the TB can store any value it likes.
If there is a drawing capture, the position is either won or drawn. If it is won, the TB has to store a win value. If the position is drawn, the TB stores either a draw or a loss value. The probing code knows that a loss in the TB means a draw (because it will see the drawing capture).
(With cursed wins and blessed losses in the mix things get still more complicated, but the principle stays the same.)
The problem with doing this for in-check position is that the probing code would have to probe a number of 6-piece positions to know the value of a 6-piece in-check position. Limiting the trick to captures, the probing code only needs to probe 5-piece positions (and the actual 6-piece position being probed). Probing 5-piece TBs should normally be fast because those files are much smaller and will quickly be cached into RAM.
In principle a TB can be generated on the fly if you don't care about access speed (and memory usage of the probing code), so there is no real lower limit to the size of the compressed TB.
2) How much of a pain would it be to make them perfect in the sense of adding en passant and castling cases?
They contain en passant in the sense that the probing code will return the correct results.
Adding castling requires more code than it is worth (to me), but the generator code is available for anyone who wants to try.
Getting en passant right is absolutely critical for TBs to be useful for game play. Getting castling right is absolutely non-critical.