7-men Syzygy attempt

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Sesse
Posts: 160
Joined: Mon Apr 30, 2018 9:51 pm
Contact:

Re: 7-men Syzygy attempt

Post by Sesse » Sun Sep 09, 2018 4:27 pm

rsync is nice, but very CPU-heavy, so it doesn't scale well for busy servers at all. There are variations like zsync that try to remedy some of this by not requiring an active server that checksums lots of data, but I don't think they ever caught on (and zsync requires extra storage space).

I couldn't find any checksums on your FTP server; did I miss anything?

noobpwnftw
Posts: 275
Joined: Sun Nov 08, 2015 10:10 pm

Re: 7-men Syzygy attempt

Post by noobpwnftw » Sun Sep 09, 2018 7:32 pm

Sesse wrote:
Sun Sep 09, 2018 4:27 pm
I couldn't find any checksums on your FTP server; did I miss anything?
Wise FTP server not serving the file unless explicitly labeled by SELinux :D

Now it is fixed, ftp://ftp.chessdb.cn/pub/syzygy/7men/checksums.txt

Sesse
Posts: 160
Joined: Mon Apr 30, 2018 9:51 pm
Contact:

Re: 7-men Syzygy attempt

Post by Sesse » Sun Sep 09, 2018 7:51 pm

All my files match that checksum file. Success!

(This is after I replaced the faulty KRRNvKRR.rtbz earlier today, of course.)

jkominek
Posts: 4
Joined: Tue Sep 04, 2018 3:33 am
Full name: John Kominek

Re: 7-men Syzygy attempt

Post by jkominek » Mon Sep 10, 2018 5:30 am

On my end I believe I now have a complete and correct set. On your side as you (Sesse) noticed, you had an old version of KRRNvKRR.rtbz, the give-away being the file size discrepancy compared to the lichess ftp listing.

The remaining five -- KNNPPvKR.rtbw, KQBNvKQN.rtbw, KQRPvKRB.rtbw, KRBBNvKQ.rtbz, KRNPvKQN.rtbz -- were file corruptions on my side. Even though they showed the correct file length they did not transfer correctly.

Question for Sesse. When you said that your set passes tbcheck, did you do the full check wherein the hash sum is recomputed and compared to the stored value, or did you do the quick check? The quick check only proves that the metadata block containing the sum transferred correctly. It won't catch a bad bit inside the data payload.

Demonstration...

Code: Select all

-rw-r--r-- 1 jkominek jkominek         240 Sep 10 00:22 internal_checksums_bad5.txt
-rw-r--r-- 1 jkominek jkominek 14609178768 Sep  9 23:58 KNNPPvKR.rtbw
-rw-r--r-- 1 jkominek jkominek 28587231440 Sep 10 00:03 KQBNvKQN.rtbw
-rw-r--r-- 1 jkominek jkominek 10277262928 Sep 10 00:20 KQRPvKRB.rtbw
-rw-r--r-- 1 jkominek jkominek 29043086864 Sep  9 23:54 KRBBNvKQ.rtbz
-rw-r--r-- 1 jkominek jkominek 78536280464 Sep 10 00:16 KRNPvKQN.rtbz
jkominek@home2:/mnt/data1/data/bases/syzygy/bad$ cat internal_checksums_bad5.txt 
KNNPPvKR.rtbw: 606c1b7c114ae9a8723f22b20ea0d1b8
KQBNvKQN.rtbw: ee56fdb347cb38d16048b2c123d814f1
KQRPvKRB.rtbw: e0214ab39644b5def7b3ff9669d09d60
KRBBNvKQ.rtbz: f0d7f89222062469a8bd1918e55472e9
KRNPvKQN.rtbz: 1514ead8b401c03841a2390379dfbcc2
jkominek@home2:/mnt/data1/data/bases/syzygy/bad$ ../tbcheck --compare internal_checksums_bad5.txt 
KNNPPvKR.rtbw: OK!
KQBNvKQN.rtbw: OK!
KQRPvKRB.rtbw: OK!
KRBBNvKQ.rtbz: OK!
KRNPvKQN.rtbz: OK!
jkominek@home2:/mnt/data1/data/bases/syzygy/bad$ ../tbcheck K*
KNNPPvKR.rtbw: FAIL!
KQBNvKQN.rtbw: FAIL!
KQRPvKRB.rtbw: FAIL!
KRBBNvKQ.rtbz: FAIL!
KRNPvKQN.rtbz: FAIL!
("bad5" refers to my 5 bad files, not that the internal checksums provided by noobpwnftw are bad.)

One reason I'm an advocate of using a common external checksum algorithm such as the sha family is their wide availability. The programs sha256sum and the older md5sum are "just there" on most Linux distributions. Presumably for Windows and Mac, too. As it happens syzygy would not compile on my two home machines straight out of the git repo. It took me a while to find a compiler version and flag settings combination that work.

Thank you for your help in cross-checking. I'm in the process of one final full sweep of the files. No errors so far.

jk

jkominek
Posts: 4
Joined: Tue Sep 04, 2018 3:33 am
Full name: John Kominek

Re: 7-men Syzygy attempt

Post by jkominek » Mon Sep 10, 2018 5:37 am

A one-letter typo in the syzygy codebase: sizt_t

Code: Select all

util.c, line 303:
static void decompress(struct CompressState *state, void *dst, sizt_t chunk,

This branch must not be covered in the regression tests. The line is conditional compiled out by the makefile as distributed, i.e. -DUSE_ZSTD invokes the zstd library instead of lz4, which the quoted line is a part of.

Sesse
Posts: 160
Joined: Mon Apr 30, 2018 9:51 pm
Contact:

Re: 7-men Syzygy attempt

Post by Sesse » Mon Sep 10, 2018 7:19 am

jkominek wrote:
Mon Sep 10, 2018 5:30 am
Question for Sesse. When you said that your set passes tbcheck, did you do the full check wherein the hash sum is recomputed and compared to the stored value, or did you do the quick check? The quick check only proves that the metadata block containing the sum transferred correctly. It won't catch a bad bit inside the data payload.
For every file I downloaded, I ran tbcheck (with no parameters) on it before I made it public for download. Thus, it checks that the file is self-consistent (internal checksum matches actual data). After the reference checksum file was posted here yesterday, I did a quick compare against that (ie., internal checksum matches external checksum).

Sesse
Posts: 160
Joined: Mon Apr 30, 2018 9:51 pm
Contact:

Re: 7-men Syzygy attempt

Post by Sesse » Wed Sep 12, 2018 9:54 pm

So with the dust settling from the generation, where are we? Stockfish master has full support; is there a version of Fathom with 7-man support yet?

User avatar
yurikvelo
Posts: 467
Joined: Sat Dec 06, 2014 12:53 pm

Re: 7-men Syzygy attempt

Post by yurikvelo » Tue Oct 02, 2018 7:33 am

Is uTorrent/BitTortent compatible with this file set?

Sesse
Posts: 160
Joined: Mon Apr 30, 2018 9:51 pm
Contact:

Re: 7-men Syzygy attempt

Post by Sesse » Wed Oct 03, 2018 7:15 pm

There's no active 7-man torrent that I know of. Of course, nothing would stop you from making one and uploading it to a tracker (but then again, it's unlikely you would find too many people interested in seeding).

khunpon
Posts: 1
Joined: Thu Oct 04, 2018 3:29 am
Full name: Khunpon Kaewdang

Re: 7-men Syzygy attempt

Post by khunpon » Sat Oct 13, 2018 7:51 am

noobpwnftw wrote:
Sun Sep 09, 2018 7:32 pm
Sesse wrote:
Sun Sep 09, 2018 4:27 pm
I couldn't find any checksums on your FTP server; did I miss anything?
Wise FTP server not serving the file unless explicitly labeled by SELinux :D

Now it is fixed, ftp://ftp.chessdb.cn/pub/syzygy/7men/checksums.txt
Hi, I can't access your ftp from Thailand.
Do you interesting todo thai-chess too?

Regards,
Pond

Post Reply