7-men Syzygy attempt

[Sesse]

rsync is nice, but very CPU-heavy, so it doesn't scale well for busy servers at all. There are variations like zsync that try to remedy some of this by not requiring an active server that checksums lots of data, but I don't think they ever caught on (and zsync requires extra storage space).

I couldn't find any checksums on your FTP server; did I miss anything?

[noobpwnftw]

I couldn't find any checksums on your FTP server; did I miss anything?

Wise FTP server not serving the file unless explicitly labeled by SELinux

Now it is fixed, ftp://ftp.chessdb.cn/pub/syzygy/7men/checksums.txt

[Sesse]

All my files match that checksum file. Success!

(This is after I replaced the faulty KRRNvKRR.rtbz earlier today, of course.)

[jkominek]

On my end I believe I now have a complete and correct set. On your side as you (Sesse) noticed, you had an old version of KRRNvKRR.rtbz, the give-away being the file size discrepancy compared to the lichess ftp listing.

The remaining five -- KNNPPvKR.rtbw, KQBNvKQN.rtbw, KQRPvKRB.rtbw, KRBBNvKQ.rtbz, KRNPvKQN.rtbz -- were file corruptions on my side. Even though they showed the correct file length they did not transfer correctly.

Question for Sesse. When you said that your set passes tbcheck, did you do the full check wherein the hash sum is recomputed and compared to the stored value, or did you do the quick check? The quick check only proves that the metadata block containing the sum transferred correctly. It won't catch a bad bit inside the data payload.

Demonstration...

Code: Select all

-rw-r--r-- 1 jkominek jkominek         240 Sep 10 00:22 internal_checksums_bad5.txt
-rw-r--r-- 1 jkominek jkominek 14609178768 Sep  9 23:58 KNNPPvKR.rtbw
-rw-r--r-- 1 jkominek jkominek 28587231440 Sep 10 00:03 KQBNvKQN.rtbw
-rw-r--r-- 1 jkominek jkominek 10277262928 Sep 10 00:20 KQRPvKRB.rtbw
-rw-r--r-- 1 jkominek jkominek 29043086864 Sep  9 23:54 KRBBNvKQ.rtbz
-rw-r--r-- 1 jkominek jkominek 78536280464 Sep 10 00:16 KRNPvKQN.rtbz
jkominek@home2:/mnt/data1/data/bases/syzygy/bad$ cat internal_checksums_bad5.txt 
KNNPPvKR.rtbw: 606c1b7c114ae9a8723f22b20ea0d1b8
KQBNvKQN.rtbw: ee56fdb347cb38d16048b2c123d814f1
KQRPvKRB.rtbw: e0214ab39644b5def7b3ff9669d09d60
KRBBNvKQ.rtbz: f0d7f89222062469a8bd1918e55472e9
KRNPvKQN.rtbz: 1514ead8b401c03841a2390379dfbcc2
jkominek@home2:/mnt/data1/data/bases/syzygy/bad$ ../tbcheck --compare internal_checksums_bad5.txt 
KNNPPvKR.rtbw: OK!
KQBNvKQN.rtbw: OK!
KQRPvKRB.rtbw: OK!
KRBBNvKQ.rtbz: OK!
KRNPvKQN.rtbz: OK!
jkominek@home2:/mnt/data1/data/bases/syzygy/bad$ ../tbcheck K*
KNNPPvKR.rtbw: FAIL!
KQBNvKQN.rtbw: FAIL!
KQRPvKRB.rtbw: FAIL!
KRBBNvKQ.rtbz: FAIL!
KRNPvKQN.rtbz: FAIL!

("bad5" refers to my 5 bad files, not that the internal checksums provided by noobpwnftw are bad.)

One reason I'm an advocate of using a common external checksum algorithm such as the sha family is their wide availability. The programs sha256sum and the older md5sum are "just there" on most Linux distributions. Presumably for Windows and Mac, too. As it happens syzygy would not compile on my two home machines straight out of the git repo. It took me a while to find a compiler version and flag settings combination that work.

Thank you for your help in cross-checking. I'm in the process of one final full sweep of the files. No errors so far.

jk

[jkominek]

A one-letter typo in the syzygy codebase: sizt_t

Code: Select all

util.c, line 303:
static void decompress(struct CompressState *state, void *dst, sizt_t chunk,

This branch must not be covered in the regression tests. The line is conditional compiled out by the makefile as distributed, i.e. -DUSE_ZSTD invokes the zstd library instead of lz4, which the quoted line is a part of.

[Sesse]

Question for Sesse. When you said that your set passes tbcheck, did you do the full check wherein the hash sum is recomputed and compared to the stored value, or did you do the quick check? The quick check only proves that the metadata block containing the sum transferred correctly. It won't catch a bad bit inside the data payload.

For every file I downloaded, I ran tbcheck (with no parameters) on it before I made it public for download. Thus, it checks that the file is self-consistent (internal checksum matches actual data). After the reference checksum file was posted here yesterday, I did a quick compare against that (ie., internal checksum matches external checksum).

[Sesse]

So with the dust settling from the generation, where are we? Stockfish master has full support; is there a version of Fathom with 7-man support yet?

[yurikvelo]

Is uTorrent/BitTortent compatible with this file set?

[Sesse]

There's no active 7-man torrent that I know of. Of course, nothing would stop you from making one and uploading it to a tracker (but then again, it's unlikely you would find too many people interested in seeding).

[khunpon]

I couldn't find any checksums on your FTP server; did I miss anything?

Wise FTP server not serving the file unless explicitly labeled by SELinux

Now it is fixed, ftp://ftp.chessdb.cn/pub/syzygy/7men/checksums.txt

Hi, I can't access your ftp from Thailand.
Do you interesting todo thai-chess too?

Regards,
Pond