rsync is nice, but very CPU-heavy, so it doesn't scale well for busy servers at all. There are variations like zsync that try to remedy some of this by not requiring an active server that checksums lots of data, but I don't think they ever caught on (and zsync requires extra storage space).
I couldn't find any checksums on your FTP server; did I miss anything?
7-men Syzygy attempt
Moderators: hgm, chrisw, Rebel
-
- Posts: 694
- Joined: Sun Nov 08, 2015 11:10 pm
- Full name: Bojun Guo
Re: 7-men Syzygy attempt
Wise FTP server not serving the file unless explicitly labeled by SELinux
Now it is fixed, ftp://ftp.chessdb.cn/pub/syzygy/7men/checksums.txt
-
- Posts: 300
- Joined: Mon Apr 30, 2018 11:51 pm
Re: 7-men Syzygy attempt
All my files match that checksum file. Success!
(This is after I replaced the faulty KRRNvKRR.rtbz earlier today, of course.)
(This is after I replaced the faulty KRRNvKRR.rtbz earlier today, of course.)
-
- Posts: 55
- Joined: Tue Sep 04, 2018 5:33 am
- Full name: John Kominek
Re: 7-men Syzygy attempt
On my end I believe I now have a complete and correct set. On your side as you (Sesse) noticed, you had an old version of KRRNvKRR.rtbz, the give-away being the file size discrepancy compared to the lichess ftp listing.
The remaining five -- KNNPPvKR.rtbw, KQBNvKQN.rtbw, KQRPvKRB.rtbw, KRBBNvKQ.rtbz, KRNPvKQN.rtbz -- were file corruptions on my side. Even though they showed the correct file length they did not transfer correctly.
Question for Sesse. When you said that your set passes tbcheck, did you do the full check wherein the hash sum is recomputed and compared to the stored value, or did you do the quick check? The quick check only proves that the metadata block containing the sum transferred correctly. It won't catch a bad bit inside the data payload.
Demonstration...
("bad5" refers to my 5 bad files, not that the internal checksums provided by noobpwnftw are bad.)
One reason I'm an advocate of using a common external checksum algorithm such as the sha family is their wide availability. The programs sha256sum and the older md5sum are "just there" on most Linux distributions. Presumably for Windows and Mac, too. As it happens syzygy would not compile on my two home machines straight out of the git repo. It took me a while to find a compiler version and flag settings combination that work.
Thank you for your help in cross-checking. I'm in the process of one final full sweep of the files. No errors so far.
jk
The remaining five -- KNNPPvKR.rtbw, KQBNvKQN.rtbw, KQRPvKRB.rtbw, KRBBNvKQ.rtbz, KRNPvKQN.rtbz -- were file corruptions on my side. Even though they showed the correct file length they did not transfer correctly.
Question for Sesse. When you said that your set passes tbcheck, did you do the full check wherein the hash sum is recomputed and compared to the stored value, or did you do the quick check? The quick check only proves that the metadata block containing the sum transferred correctly. It won't catch a bad bit inside the data payload.
Demonstration...
Code: Select all
-rw-r--r-- 1 jkominek jkominek 240 Sep 10 00:22 internal_checksums_bad5.txt
-rw-r--r-- 1 jkominek jkominek 14609178768 Sep 9 23:58 KNNPPvKR.rtbw
-rw-r--r-- 1 jkominek jkominek 28587231440 Sep 10 00:03 KQBNvKQN.rtbw
-rw-r--r-- 1 jkominek jkominek 10277262928 Sep 10 00:20 KQRPvKRB.rtbw
-rw-r--r-- 1 jkominek jkominek 29043086864 Sep 9 23:54 KRBBNvKQ.rtbz
-rw-r--r-- 1 jkominek jkominek 78536280464 Sep 10 00:16 KRNPvKQN.rtbz
jkominek@home2:/mnt/data1/data/bases/syzygy/bad$ cat internal_checksums_bad5.txt
KNNPPvKR.rtbw: 606c1b7c114ae9a8723f22b20ea0d1b8
KQBNvKQN.rtbw: ee56fdb347cb38d16048b2c123d814f1
KQRPvKRB.rtbw: e0214ab39644b5def7b3ff9669d09d60
KRBBNvKQ.rtbz: f0d7f89222062469a8bd1918e55472e9
KRNPvKQN.rtbz: 1514ead8b401c03841a2390379dfbcc2
jkominek@home2:/mnt/data1/data/bases/syzygy/bad$ ../tbcheck --compare internal_checksums_bad5.txt
KNNPPvKR.rtbw: OK!
KQBNvKQN.rtbw: OK!
KQRPvKRB.rtbw: OK!
KRBBNvKQ.rtbz: OK!
KRNPvKQN.rtbz: OK!
jkominek@home2:/mnt/data1/data/bases/syzygy/bad$ ../tbcheck K*
KNNPPvKR.rtbw: FAIL!
KQBNvKQN.rtbw: FAIL!
KQRPvKRB.rtbw: FAIL!
KRBBNvKQ.rtbz: FAIL!
KRNPvKQN.rtbz: FAIL!
One reason I'm an advocate of using a common external checksum algorithm such as the sha family is their wide availability. The programs sha256sum and the older md5sum are "just there" on most Linux distributions. Presumably for Windows and Mac, too. As it happens syzygy would not compile on my two home machines straight out of the git repo. It took me a while to find a compiler version and flag settings combination that work.
Thank you for your help in cross-checking. I'm in the process of one final full sweep of the files. No errors so far.
jk
-
- Posts: 55
- Joined: Tue Sep 04, 2018 5:33 am
- Full name: John Kominek
Re: 7-men Syzygy attempt
A one-letter typo in the syzygy codebase: sizt_t
This branch must not be covered in the regression tests. The line is conditional compiled out by the makefile as distributed, i.e. -DUSE_ZSTD invokes the zstd library instead of lz4, which the quoted line is a part of.
Code: Select all
util.c, line 303:
static void decompress(struct CompressState *state, void *dst, sizt_t chunk,
This branch must not be covered in the regression tests. The line is conditional compiled out by the makefile as distributed, i.e. -DUSE_ZSTD invokes the zstd library instead of lz4, which the quoted line is a part of.
-
- Posts: 300
- Joined: Mon Apr 30, 2018 11:51 pm
Re: 7-men Syzygy attempt
For every file I downloaded, I ran tbcheck (with no parameters) on it before I made it public for download. Thus, it checks that the file is self-consistent (internal checksum matches actual data). After the reference checksum file was posted here yesterday, I did a quick compare against that (ie., internal checksum matches external checksum).jkominek wrote: ↑Mon Sep 10, 2018 7:30 am Question for Sesse. When you said that your set passes tbcheck, did you do the full check wherein the hash sum is recomputed and compared to the stored value, or did you do the quick check? The quick check only proves that the metadata block containing the sum transferred correctly. It won't catch a bad bit inside the data payload.
-
- Posts: 300
- Joined: Mon Apr 30, 2018 11:51 pm
Re: 7-men Syzygy attempt
So with the dust settling from the generation, where are we? Stockfish master has full support; is there a version of Fathom with 7-man support yet?
-
- Posts: 710
- Joined: Sat Dec 06, 2014 1:53 pm
Re: 7-men Syzygy attempt
Is uTorrent/BitTortent compatible with this file set?
-
- Posts: 300
- Joined: Mon Apr 30, 2018 11:51 pm
Re: 7-men Syzygy attempt
There's no active 7-man torrent that I know of. Of course, nothing would stop you from making one and uploading it to a tracker (but then again, it's unlikely you would find too many people interested in seeding).
-
- Posts: 1
- Joined: Thu Oct 04, 2018 5:29 am
- Full name: Khunpon Kaewdang
Re: 7-men Syzygy attempt
Hi, I can't access your ftp from Thailand.noobpwnftw wrote: ↑Sun Sep 09, 2018 9:32 pmWise FTP server not serving the file unless explicitly labeled by SELinux
Now it is fixed, ftp://ftp.chessdb.cn/pub/syzygy/7men/checksums.txt
Do you interesting todo thai-chess too?
Regards,
Pond