7-men Syzygy attempt

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

Vinvin
Posts: 5239
Joined: Thu Mar 09, 2006 9:40 am
Full name: Vincent Lejeune

Re: 7-men Syzygy attempt

Post by Vinvin »

Dann Corbit wrote:...
I think it would be fun to have my very own 7 man file set. I have a great big NAS cabinet, and I can just add on 12 TB disks as they get built. I guess that storage costs will continue to fall over time so it may not be as expensive as one might think.
And Intel Optane is coming : https://www.anandtech.com/show/12512/th ... p-review/5 8-)
Dann Corbit
Posts: 12662
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: 7-men Syzygy attempt

Post by Dann Corbit »

phhnguyen wrote:
Dann Corbit wrote:
noobpwnftw wrote:About the "find_val:not found" problem happened at the third reduction, I think this line may need to be changed into

Code: Select all

j = CAPT_CWIN_RED + 1 + i - REDUCE_PLY_RED;
I'm still not quite sure about this.
Once you get the code fixed, I can do some runs on a high end machine if you like. I just bought a bunch of SSDs including 2x2TB and 5 smaller ones.

I think it would be fun to have my very own 7 man file set. I have a great big NAS cabinet, and I can just add on 12 TB disks as they get built. I guess that storage costs will continue to fall over time so it may not be as expensive as one might think.
Look like you have been solving storage problem for 7 men!!!

How about RAM? Will you soon have 1 TB? ;)
I have two machines with 128 GB, but 1TB RAM is a lot of cash right now
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
Dann Corbit
Posts: 12662
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: 7-men Syzygy attempt

Post by Dann Corbit »

syzygy wrote:
Nordlandia wrote:Concerning DTM bases. Have you checked whether placing them on regular hard disk (HDD) inflict/cause noticeabe performance drop. Similarly like dtz works fine on HDD without noticeable performance penalty.
HDD is fine if they are probed only at the root (although finding the whole winning line can take a while).

Probing DTM during the search really needs them to be on SSD.
Two Terabyte SSD drives can be bought for $360 now. The prices are finally starting to drop again.

Those super-high speed gumstick SSDs can be had for abouit $400 per Terabyte. Probably not necessary for using the SSD files, but I bet they are great for building them.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
User avatar
Nordlandia
Posts: 2821
Joined: Fri Sep 25, 2015 9:38 pm
Location: Sortland, Norway

Re: 7-men Syzygy attempt

Post by Nordlandia »

Maybe PCIe or NVMe types of SSDs is preferable for 7-man syzygy.

I'm worried that regular consumer SSDs cam reduce engine speed if probing 7-man. Probing full 7-man set can inflict moderate speed penalty compared to low/slight speed penalty of 6-man.

Dann Corbit: if Ronald's 500 GB estimation for DTM bases is correct, 1 TB SSD is sufficient enough for syzygy tablebases.

500 + 150 = 650. 350 GB available for something else.
Dann Corbit
Posts: 12662
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: 7-men Syzygy attempt

Post by Dann Corbit »

Nordlandia wrote:Maybe PCIe or NVMe types of SSDs is preferable for 7-man syzygy.

I'm worried that regular consumer SSDs cam reduce engine speed if probing 7-man. Probing full 7-man set can inflict moderate speed penalty compared to low/slight speed penalty of 6-man.

Dann Corbit: if Ronald's 500 GB estimation for DTM bases is correct, 1 TB SSD is sufficient enough for syzygy tablebases.

500 + 150 = 650. 350 GB available for something else.
He is working on DTM. If those fit on 500GB for 7 men, I will be more than a little bit surprised. In fact astonished is not enough.

I also guess that the win from a 7 man probe hit is much greater than 6 man, which is already better than 5 man.

In ten years a TB of RAM will be affordable on desktop PCs for the average consumer.

And if the probes cost too much, then reduce the fraction of times that you probe. For instance, probing only at the root is cheap, and probing only at the root and at pv nodes is also cheap.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
Vinvin
Posts: 5239
Joined: Thu Mar 09, 2006 9:40 am
Full name: Vincent Lejeune

Re: 7-men Syzygy attempt

Post by Vinvin »

Vinvin wrote:
Dann Corbit wrote:...
I think it would be fun to have my very own 7 man file set. I have a great big NAS cabinet, and I can just add on 12 TB disks as they get built. I guess that storage costs will continue to fall over time so it may not be as expensive as one might think.
And Intel Optane is coming : https://www.anandtech.com/show/12512/th ... p-review/5 8-)
I just found a video about new Optane products here :
noobpwnftw
Posts: 684
Joined: Sun Nov 08, 2015 11:10 pm
Full name: Bojun Guo

Re: 7-men Syzygy attempt

Post by noobpwnftw »

I did change the stat limits according to their pawn version like so.

Now MAX_PLY=768 should be sufficient for 7-men TBs, I will verify the generation process works without problem before moving on to the compression part.
syzygy
Posts: 5647
Joined: Tue Feb 28, 2012 11:56 pm

Re: 7-men Syzygy attempt

Post by syzygy »

noobpwnftw wrote:I did change the stat limits according to their pawn version like so.

Now MAX_PLY=768 should be sufficient for 7-men TBs, I will verify the generation process works without problem before moving on to the compression part.
I will add my 16-bit permutation and compression code to github, but I will probably not have time before the weekend.

The DTZ format has to be adapted to allow 16-bit values in the "map"' array. It would be possible to do this just for those tables that need it, so the tables already generated remain valid.

For pawns it might be good to make a bigger change: compressing data per rank instead of per file. That might lead to somewhat better compression, but it is also a bit more complicated and it will need some amount of programming. I have done this for DTM, but in a kind of brute force way (generating the whole table at once, which is not an option for 7 pieces). I guess the generation part can still be done per file.
Ras
Posts: 2582
Joined: Tue Aug 30, 2016 8:19 pm
Full name: Rasmus Althoff

Re: 7-men Syzygy attempt

Post by Ras »

Btw, in tbcore.c open_tb(), I think there is a buffer overflow waiting to happen when copying over the path names.
noobpwnftw
Posts: 684
Joined: Sun Nov 08, 2015 11:10 pm
Full name: Bojun Guo

Re: 7-men Syzygy attempt

Post by noobpwnftw »

I will add my 16-bit permutation and compression code to github, but I will probably not have time before the weekend.
Thank you, and I think it is possible to differentiate between file formats with the fifth byte in the header or just use another magic value.

With your new 16-bit compression code available, I'm pretty confident that full set of 7-men Syzygy will be available soon.

I already have the necessary hardware(both steps in physical RAM without --disk) and will provide hosting for all the resulting files.