Syzygy EGTB's via Torrent Thread

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Kirill Kryukov
Posts: 492
Joined: Sun Mar 19, 2006 3:12 am

Re: Syzygy EGTB's via Torrent Thread

Post by Kirill Kryukov » Wed Sep 11, 2013 2:35 pm

jshriver wrote:
Ozymandias wrote:Will you be making separate torrents for the WDL and DTZ sets?
Certainly hope so.
Actually I hadn't thought of that. At the moment a torrent is running live for the entire set.

Can understand the reasoning for this, but at the same time, the more people we have going into it the faster it'll be for everyone. So fragmenting it would reduce speed.
If someone wants just WDL (due to bandwidth or storage restrictions), his only option currently is to click on complete torrent, and then manually uncheck DTZ files one by one. Not a really satisfactory solution for this very likely scenario.

Same for 3-4-5 and 6-piece sets. Duplicating 3-4-5 both as separate torrent and as part of 3-4-5-6 collection results in waste of bandwidth. Say, person A downloads 3-4-5 set, person B downloads the whole 3-4-5-6 set from second torrent. When person C tries to download 3-4-5 set, he only downloads from A, although person B keeps seeding the whole 3-4-5-6 set. Waste of bandwidth.

On the other hand, offering only 3-4-5-6 collection (not 3-4-5 one), while removes the duplication, creates difficulties for those who only need 3-4-5 files. So the way I see it the solution may be to offer 3-4-5-piece set and 6-piece set, but not 3-4-5-6 together. (Adding WDL / DTZ split this means 4 torrents).

User avatar
jshriver
Posts: 1105
Joined: Wed Mar 08, 2006 8:41 pm
Location: Morgantown, WV, USA
Contact:

Re: Syzygy EGTB's via Torrent Thread

Post by jshriver » Wed Sep 11, 2013 2:40 pm

Sounds like a good idea just removed it from the tracker.

It was already up and running before this was recommended. :)

Will take a lil time to sort the files out, but will get them up hopefully today.

-Josh

Edit: I'm going to leave the 3-4-5 set up as it is. It's nice if someone wants to get just that set.

From the test I ran during the night, with only 1 seed and 2 leechers near the end one of them was already up to 1.1 megs a second and both finished fairly quickly.

I will separate out the 6 men by WDL/DTZ though.

User avatar
Ozymandias
Posts: 1125
Joined: Sun Oct 25, 2009 12:30 am

Re: Syzygy EGTB's via Torrent Thread

Post by Ozymandias » Wed Sep 11, 2013 2:56 pm

Thank you very much, sorry I didn't suggest this earlier. I'll try to make up for it by seeding well past the 1:1 ratio.

User avatar
jshriver
Posts: 1105
Joined: Wed Mar 08, 2006 8:41 pm
Location: Morgantown, WV, USA
Contact:

Re: Syzygy EGTB's via Torrent Thread

Post by jshriver » Wed Sep 11, 2013 4:09 pm

Ozymandias wrote:Thank you very much, sorry I didn't suggest this earlier. I'll try to make up for it by seeding well past the 1:1 ratio.
No problem, it was a great suggestion from both of you. Plus from my understanding the probing code doesn't hit the DTZ till a certain point anyway so WDL are the most crucial.

The 6 Men WDL are now up, and will be seeded shortly.

Aiming to have DTZ up and seeded today as well. Appreciate everyone's patience.

User avatar
Ozymandias
Posts: 1125
Joined: Sun Oct 25, 2009 12:30 am

Re: Syzygy EGTB's via Torrent Thread

Post by Ozymandias » Wed Sep 11, 2013 6:14 pm

Ok, I'll start freeing up disk space ;-)

User avatar
jshriver
Posts: 1105
Joined: Wed Mar 08, 2006 8:41 pm
Location: Morgantown, WV, USA
Contact:

Re: Syzygy EGTB's via Torrent Thread

Post by jshriver » Wed Sep 11, 2013 7:29 pm

Well the entire set is up now and seems to be running solid. Even with only 1 seed right now, it went up from 20-30k to now about 1/2 meg. As more people join in it'll only get faster.

Enjoy! Let me know if there are any problems or concerns. I made sure to run a verify right before creating the torrents to ensure data quality.

-Josh

syzygy
Posts: 4455
Joined: Tue Feb 28, 2012 10:56 pm

Re: Syzygy EGTB's via Torrent Thread

Post by syzygy » Wed Sep 11, 2013 8:28 pm

jshriver wrote:Hoping within the week to get the full 151gig 6men set up via this method.
Before you make them available, please make sure the generation was fully correct:

Code: Select all

tbcheck --compare checksums/wdl345.txt
tbcheck --compare checksums/dtz345.txt
tbcheck --compare checksums/wdl6.txt
tbcheck --compare checksums/dtz6.txt
This is very quick, because it only checks whether the checksum embedded in each file is the expected checksum.

To verify integrity after download, either your md5sums can be used, or tbcheck:

Code: Select all

tbcheck *.rtbw *.rtbz
This takes a bit more time (but should be faster than running md5sum), because it calculates the checksum of each file and compares it to the embedded checksum.

User avatar
jshriver
Posts: 1105
Joined: Wed Mar 08, 2006 8:41 pm
Location: Morgantown, WV, USA
Contact:

Re: Syzygy EGTB's via Torrent Thread

Post by jshriver » Wed Sep 11, 2013 8:30 pm

I ran the built in tbcheck -verify on the complete set earlier. So they are clean. :) Thank you for making this.

syzygy
Posts: 4455
Joined: Tue Feb 28, 2012 10:56 pm

Re: Syzygy EGTB's via Torrent Thread

Post by syzygy » Wed Sep 11, 2013 8:38 pm

jshriver wrote:
Ozymandias wrote:Thank you very much, sorry I didn't suggest this earlier. I'll try to make up for it by seeding well past the 1:1 ratio.
No problem, it was a great suggestion from both of you. Plus from my understanding the probing code doesn't hit the DTZ till a certain point anyway so WDL are the most crucial.
In principle the WDL files ("bitbases") can indeed be used without the DTZ files. On the other hand the DTZ files are useless without the WDL files.

I think it is a good idea to separately distribute WDL345, DTZ345, WDL6 and DTZ6.

I would recommend users to store WDL files on SSD if sufficient SSD space is available. There is no need to store DTZ files on SSD, they can be kept on a regular HD.

syzygy
Posts: 4455
Joined: Tue Feb 28, 2012 10:56 pm

Re: Syzygy EGTB's via Torrent Thread

Post by syzygy » Wed Sep 11, 2013 8:39 pm

jshriver wrote:I ran the built in tbcheck -verify on the complete set earlier. So they are clean. :) Thank you for making this.
Please do run tbcheck --compare. It just takes a second. This is not the same as tbverify.

Post Reply