EGTB compression

Discussion of chess software programming and technical issues.

Moderators: Harvey Williamson, bob, hgm

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
Dann Corbit
Posts: 11626
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

EGTB compression

Post by Dann Corbit » Wed Oct 14, 2020 5:54 pm

Latest version of LC4 using HC -9 gives a compression ratio of 2.721:1, and while it only compresses at 41 MB/s it decompresses at 4900 MB/s (1/3 the speed of memcpy). Since you only have to compress them once, the compression speed is pretty well irrelevant.

Seems likely that Ronald has already implemented this, but other systems like Gaviota TB and Nalimov TB might make advantage of it.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

syzygy
Posts: 4807
Joined: Tue Feb 28, 2012 10:56 pm

Re: EGTB compression

Post by syzygy » Thu Oct 15, 2020 11:50 pm

I guess you mean LZ4, not LC4.

My generator includes LZ4 compression code, but only for compressing intermediate files during generation. This is down at the lowest compression setting (I think) because otherwise it takes too long. The point of the compressing the intermediate files is to speed up the generation process.

Some time ago I switched the default compression code from LZ4 to Facebook's ZSTD, which gave much better compression at fast settings (if I remember well).

The final TB files are compressed with a custom compression method that is unrelated to any general purpose compressor.

Dann Corbit
Posts: 11626
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: EGTB compression

Post by Dann Corbit » Fri Oct 16, 2020 11:00 pm

I see.
Yes, I meant LZ4.
With the HC version using -9, it gets very good compression ratio. Now, it compresses slowly using those settings but it decompresses ultra-fast.
That is why I thought it would be a good compression algorithm for tablebase files. Even though compression is slow, you only have to compress the files once. There is a very good compression ratio, and the decompression is absurdly fast (1/3 of memcpy speed).

Now, if you have a custom compression scheme that is tailored to the data, it may very well be better than a more generic routine,
But (for instance) Gaviota tablebase files use standardized compression schemes.
Nalimov files use something by Andrew Kadtech, but i wonder if it could be improved (since the files are fairly large).
I guess the compression method would not be ideal for formation of Syzygy files because the compression step is not fast.

Are you still working on DTM files? I have not heard anything on that subject for a while,
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

Post Reply