Syzygy tablebases

[hgm]

They have a limited number of erase-write cycles, not? The number is large, but not infinite.

[andytl755]

Thank you very much for that.

[syzygy]

They have a limited number of erase-write cycles, not? The number is large, but not infinite.

I think max 1000 erase-write cycles is typical nowadays for consumer SSDs.

Overwriting the same small file 1000 times is not going to be a problem, because the SSD firmware will not write the data 1000 times to the same block but will spread out the writes. Effectively the limit on the number of erase-write cycles puts a limit on the total amount of data that can be written to the SSD.

WIth just a couple of GB of RAM and the random access pattern of a relatively dumb generator, each memory page will probably be swapped in and out of memory many times during each pass (perhaps with only 1 byte changed). So the total amount of data being rewritten during the generation of a single table will be huge.

[hgm]

Indeed, I don't think a dumb generator is a viable option when things get cramped. When resources are generous it is of course the obvious way to go; why work hard to leave something that is available in abundance unused? But building 7-men Pawnless EGT is a really challenging task, and it will pay off hugely even to be only slightly clever.

It is a bit like travelling to Alpha Centauri. Start the trip now, and you can be sure someone else will have been living there 1000 years to welcome you when you het thawed on arrival. Better spend 100 years developing a faster rocket first!

[Rein Halbersma]

Indeed, I don't think a dumb generator is a viable option when things get cramped. When resources are generous it is of course the obvious way to go; why work hard to leave something that is available in abundance unused? But building 7-men Pawnless EGT is a really challenging task, and it will pay off hugely even to be only slightly clever.

It is a bit like travelling to Alpha Centauri. Start the trip now, and you can be sure someone else will have been living there 1000 years to welcome you when you het thawed on arrival. Better spend 100 years developing a faster rocket first!

If we are talking about rocket science, it is currently possible to spend ~35K on a dual processor server from sites like Newegg that has 88 (hyper)cores (the Xeon E5-2699-v4) and 24 x 64 Gb of RAM (roughly 16 Gb per core), combined with ~20 Tb of HDD (two drives). It takes about 4 Moore-cycles to get from that NASA type of budget to around $2K (still expensive, but manageable by high-end consumers). Take 3 years per cycle, and you can try to launch for 7 piece dbs in roughly over a decade.

[Milos]

They have a limited number of erase-write cycles, not? The number is large, but not infinite.

Limited is an overstatement. Regular NAND cell (both 3D stacked under 45nm, or single layer under 22nm) can't survive more than 1000 write/erase cycles.
Even enterprise SSD cells can on usual survive only 5000-7000 cycles.

[Milos]

If we are talking about rocket science, it is currently possible to spend ~35K on a dual processor server from sites like Newegg that has 88 (hyper)cores (the Xeon E5-2699-v4) and 24 x 64 Gb of RAM (roughly 16 Gb per core), combined with ~20 Tb of HDD (two drives). It takes about 4 Moore-cycles to get from that NASA type of budget to around $2K (still expensive, but manageable by high-end consumers). Take 3 years per cycle, and you can try to launch for 7 piece dbs in roughly over a decade.

Why the hell would you spend ~35k for a machine???
For under 1K you can buy dual E5-2670 with 1TB of RAM. 3 such machines are much faster and better choice for generating 7-men than 44 cores E5-2699-v4 machine with 1.5TB or RAM.
P.S. Is is only 88 threads, but 44 real cores. You know E5-2xxx in Xeon marking comes from dual, meaning you can put only 2 of those on a board.

[hgm]

EGT generation is not a processing problem, but a storage problem. In my old generator, the CPU was 99% idle, waiting for the cache miss to be processed. So I doubt that multi-threading would buy you anything. NUMA parallelization of memory banks could be helpful, though.

[Rein Halbersma]

Why the hell would you spend ~35k for a machine???

Indeed! I was only saying that the most expensive dual processor machine money can currently buy is around that amount (dual 2699-v4, 1.5 Tb of DDR4-2400 RAM, pair of 10 Tb HDDs, add in a few Titan X graphics cards for machine learning etc.).

For under 1K you can buy dual E5-2670 with 1TB of RAM. 3 such machines are much faster and better choice for generating 7-men than 44 cores E5-2699-v4 machine with 1.5TB or RAM.
P.S. Is is only 88 threads, but 44 real cores. You know E5-2xxx in Xeon marking comes from dual, meaning you can put only 2 of those on a board.

I can't reproduce a 1K dual E5-2670 with 1 Tb of RAM. The two 2670s are under $200 each, but where do you get such cheap RAM? Note that the E5-2670 is limited to DDR3-1600 RAM, for which goes for $760 per 64 Gb (2 x 32 Gb) at Newegg. That's $6K for "only" 0.5 Tb and total build costs near $8K. Or are you willing to buy used sticks? Also, the SuperMicro X9D series of motherboards do not support more than 0.5 Tb anyway.

So 3 of those $8K machines is still an awful lot of money, and that is 5 year old technology, slower chipsets and RAM etc.

[syzygy]

EGT generation is not a processing problem, but a storage problem. In my old generator, the CPU was 99% idle, waiting for the cache miss to be processed. So I doubt that multi-threading would buy you anything. NUMA parallelization of memory banks could be helpful, though.

Multithreading speeds up my generator a lot even though it does not do much calculation. The memory controller clearly does not need to serialise all memory accesses.