7-men Syzygy attempt

[Sesse]

MD5 has been broken since a long time; it's known how to (fairly easily) construct two files with the same MD5 checksum. Even though this is not the same as a second preimage attack (create a file with a given MD5 sum), this makes it fairly useless as a cryptographic hash, and it has been deprecated on the web for a long time.

In the case of tablebases, maybe it doesn't matter (it's perhaps not so likely that someone would try to construct a fake/broken tablebase on purpose). But there's still no good reason to cling onto a standard everybody is trying to get rid of as soon as possible.

[jkominek]

I like sha256. It takes twice as long as md5 to generate and verify, though. Or, half the time of sha512, if you want to look at it that way.

[Sesse]

Well, twice as much CPU. If you have a few cores, you're likely to become I/O bound either way, though.

Thanks for the file; when I'm done downloading and have run tbcheck on everything, I'll verify on my side.

[syzygy]

Comparing file sizes is sufficient to ensure that your files are up to date and do not include any of the files that were found to be incorrect due to problems with the generator.

The internal checksum is sufficient to ensure that your files were not corrupted during transmission or while being stored on disc.

None of this is cryptographically secure, but that is fine as long as you are not downloading files from an unknown source. (But what are the chances that some evil hacker chooses to mess with multigigabyte tablebase files? There are easier targets than that.)

[Sesse]

Yes; it's more about eradicating MD5 from the face of the planet than genuine worry about malicious actors.

[Sesse]

tablebase.sesse.net now has the full 7-man Syzygy set. All files have been checked with tbcheck.

I'm now running sha256sum, which will take 7–8 hours or so; it's I/O limited, so the box will be quite sluggish while it's going on. After that, there's some setup and reboots to get the SSD caching back on track, but after that, it's all good.

[Sesse]

I'm not done comparing yet, but it seems there are already three mismatches.

These are the hashes I have:

Code: Select all

ec4785cf89824483c3eed88de4b8d87983f0520510c65b52ff6144ac55b744dc  KRNPvKQN.rtbz
c3bc270aa634b3e4398001e4b9e98347cfa660798fc22783sha032s1867afb47  KRRNvKRR.rtbz
6bc3fe64f334601e1c28c4770a35c065e0a9f1b736532fb49edd65da417d76ac  KRBBNvKQ.rtbz

These are the ones from your file:

Code: Select all

dac8e9f015d0677f6e54dd180934e9a2248965825d86db5064aefc7590e90a2f  KRNPvKQN.rtbz
e4f75f13faf85c9361ad689d45a7eb74b02de9f731049fc34e7adcce13fa079b  KRRNvKRR.rtbz
f4befd738eaeaac507e50379f722ce1fac71ce2bc64f6e14d1755029c82bcd80  KRBBNvKQ.rtbz

I've re-run tbcheck on these files, and all three check out OK. Do you think you could check on your side, too? These are the file sizes:

Code: Select all

-rw-r--r-- 1 sesse sesse 29043086864 Apr 16 08:14 KRBBNvKQ.rtbz
-rw-r--r-- 1 sesse sesse 78536280464 Jul 26 07:00 KRNPvKQN.rtbz
-rw-r--r-- 1 sesse sesse 11651368464 May 11 20:42 KRRNvKRR.rtbz

[Sesse]

Three mismatches in WDL, too (on different tables, actually). Again, tbcheck runs fine, so we'll need some external verification.

My hashes:

Code: Select all

275c616256ed173177ff8f68e9119335d88b6a8ef8e47ac8a8ca0b949aaf44fe  KQRPvKRB.rtbw
275d5c38fc52d25c0cd70a4ba095b13311d7bf878f2c3db41c9341d8475d8426  KNNPPvKR.rtbw
5dfd7562750f0b1873d3b91b14aa82c8690504d324a36ff76f3c2f2199dcdee7  KQBNvKQN.rtbw

Your hashes:

Code: Select all

ac57b23b8807a46059512cb1d89ed344b3a24cfd23d09350f5b1f3728262f70d  KQRPvKRB.rtbw
cfb7a50cb82ec5741b8e2f8f37610a375ed3d03f396214af05567a61668a1783  KNNPPvKR.rtbw
5f0f16789473260056c194647de4358b10296ec00838c4ab24eb48d368919e63  KQBNvKQN.rtbw

My file sizes:

Code: Select all

-rw-r--r-- 1 sesse sesse 14609178768 Jun 20 06:03 KNNPPvKR.rtbw
-rw-r--r-- 1 sesse sesse 28587231440 May 13 07:04 KQBNvKQN.rtbw
-rw-r--r-- 1 sesse sesse 10277262928 Jun 10 04:49 KQRPvKRB.rtbw

[Sesse]

I've checked file sizes against Lichess. For four of the tables, my file sizes match their mirror. For KRRNvKRR.rtbz and KRBBNvKQ.rtbz, we disagree. Both of these (as well as KQBNvKQN.rtbw, where I match with Lichess but disagree with the posted SHA256 sums) were among the files that were initially found faulty and that I had posted that I had deleted from my mirror and later re-downloaded, so I suppose I really did so. Still, in one case, I have a newer modification date than Lichess, and in another, they're newer than me. One of us has a stale file.

Let's look at 112.73.74.24, the master:

Code: Select all

-rw-r--r--    1 0        0        29043086864 Apr 16 06:14 KRBBNvKQ.rtbz
-rw-r--r--    1 0        0        78536280464 Jul 26 05:00 KRNPvKQN.rtbz
-rw-r--r--    1 0        0        8766750032 May 13 10:03 KRRNvKRR.rtbz
-rw-r--r--    1 0        0        14609178768 Jun 20 04:03 KNNPPvKR.rtbw
-rw-r--r--    1 0        0        28587231440 May 13 05:04 KQBNvKQN.rtbw
-rw-r--r--    1 0        0        10277262928 Jun 10 02:49 KQRPvKRB.rtbw

It seems KRRNvKRR.rtbz is faulty on my mirror; maybe I didn't delete the file correctly when I said I did (e.g. forgot one of the files or something). However, it agrees on the other five, in particular KRBBNvKQ.rtbz. Which suddenly now disappeared from Lichess as I'm writing?

[noobpwnftw]

It means that we really need to eradicate FTP and HTTP as they are not reliable ways to exchange large volumes of files.

I used rsync during the building process. Also I have put my tbcheck checksums on the FTP site.