What is the best way to obtain the 7-piece tablebases?

[mbabigian]

Thank you, that looks useful for Windows users. I wonder if there is something like that for Mac.

If you are not cash strapped, buy VMware's Fusion https://www.vmware.com/products/fusion.html and install Windoze in a VM. There may be some freeware Windows emulator too, but I don't do Mac or any Apple products, so I wouldn't be much help beyond the Fusion suggestion.

[Dann Corbit]

I guess that people will not seriously start on 8 man tablebase files for five more years.
Then, they will require about the same effort as the 7 man files did.

[Ajedrecista]

Hello Dann:

I guess that people will not seriously start on 8 man tablebase files for five more years.
Then, they will require about the same effort as the 7 man files did.

I agree with you. According to this site:

NULP (Number of Unique Legal Positions in chess endgames)

The number of unique legal positions (NULP) in 8-man EGTB should be circa 3.8176e+16, which is around 90.07 times the number of unique legal positions in 7-man EGTB (circa 4.2384e+14). The breakdown of NULP by endgame type is here:

https://kirill-kryukov.com/chess/nulp/r ... ndgame.txt

I downloaded an Excel with interesting EGTB statistics through 3-man to 8-man long ago, including difference in material (Q = 9, R = 5, B = 3, N = 3, P = 1), pawnful or pawnless, number of pieces by side (4 vs 4, 5 vs 3, ...). I downloaded it from CCRL Endgame Tablebase Forum and it is still downloadable:

analysis of 1 specific 6-7-8 piece position

(I found the original source thanks to an old post by myself!).

I combined some data of 8-man EGTB: NULP by endgame type, difference in material, pawnful or pawnless and number of pieces by side. Summary:

Code: Select all

                 Endgames             Positions          Percentage
===================================================================
4 vs 4 (no P)       210         1,273,144,250,260,640       3.33 %
4 vs 4 (P)          420        10,459,782,892,146,840      27.40 %
5 vs 3 (no P)       350         1,944,492,729,833,666       5.09 %
5 vs 3 (P)          700        15,851,450,424,099,256      41.52 %
6 vs 2 (no P)       224           819,877,739,413,186       2.15 %
6 vs 2 (P)          406         6,535,348,354,167,395      17.12 %
7 vs 1 (no P)        84           148,530,422,041,225       0.39 %
7 vs 1 (P)          126         1,143,680,065,786,037       3.00 %
-------------------------------------------------------------------
TOTAL              2520        38,176,306,877,748,245     100.00 %

Endgames can be ordered in many ways: by size, by difference in material and then by size...

Many of those 2520 endgames are very umbalanced. I get the following results applying filters in Excel:

Code: Select all

Side 1: UPPERCASE.
Side 2: lowercase.
#x: number of x.
Difference = ABS[9*(#Q - #q) + 5*(#R - #r) + 3*(#B - #b) + 3*(#N - #n) + 1*(#P - #p)]

Difference     Endgames
=======================
     0            157
     2            265
     4            256
     6            260
     8            229
    10            212
    12            202
    14            170
    16            145
    18            136
    20            104
    22             85
    24             75
    26             55
    28             41
    30             38
    32             24
    34             18
    36             16
    38             10
    40              6
    42              7
    44              3
    46              2
    48              2
    50              1
    52              0
    54              1
-----------------------
  TOTAL          2520

Similar analysis can be done using endgames plus side to move instead of only endgames:

https://kirill-kryukov.com/chess/nulp/r ... by-esm.txt

Regards from Spain.

Ajedrecista.

[duncan]

This article from the Lomonosov tablebase page estimates that the size of an 8-man tablebase will be ~100x larger than a 7-man table base and will take a minimum of 10,000 TB of disk space and 50 TB of RAM, limiting its completion to a top-10 super computer

Syzygy 7-piece was about 1 Tb of Ram and 18Tb hard disk

Lets say you need 100 times as much ram and hard drive for each extra piece so 9 pieces needs 10 pb Ram and 180 Pb hard drive.

This seems to be within the capability of Summit supercomputer


https://lichess.org/blog/W3WeMyQAACQAdf ... e-complete

7-piece Syzygy tablebases are complete
..

Bojun Guo (also one of the top contributors of computing power for Stockfish testing) started generating the first tables in May, expertly managing two machines, one of them with 384 threads (8x Intel Xeon Platinum 8168) and more than 1 TB of RAM (later distributed differently).

..


The number of unique legal 7-piece positions is 423,836,835,667,331. Syzygy tablebases store all their information in 18.4 TB, so at around 0.35 bits per position. This is much more compact than the proprietary 100 TB Lomonosov tablebases.


https://www.tomshardware.com/news/us-su ... 37367.html


In total, the Summit supercomputer has at is disposal 9,216 Power9 processors, over 10PB of DDR4, HBM, and non-volatile memory, 250PB of storage, and 27,648 Tesla V100 accelerators — which was certainly not a bad way to spend $200 million dollars.

[Dann Corbit]

The number of unique legal 7-piece positions is 423,836,835,667,331. Syzygy tablebases store all their information in 18.4 TB, so at around 0.35 bits per position. This is much more compact than the proprietary 100 TB Lomonosov tablebases.

But I guess that if the Syzygy tablebase format were changed to DTM, much of the space advantage would vanish.

https://www.tomshardware.com/news/us-su ... 37367.html


In total, the Summit supercomputer has at is disposal 9,216 Power9 processors, over 10PB of DDR4, HBM, and non-volatile memory, 250PB of storage, and 27,648 Tesla V100 accelerators — which was certainly not a bad way to spend $200 million dollars.

My suspicion is that while the Russian university allowed their supercomputer to be used for TB computation, the Summit supercomputer will be booked with jobs until it dies.

[Nordlandia]

Is it possible to schedule the download in chronological order. Chrome only accept one download at a time. My point is that once the download has finished for one particular table, the next start immediately, without any time wasted for nothing.

I don't know if it possible to download one from Sesse website and one from Lichess mirror concurrently ?

[swissman]

You have free online access to the 7-piece TB here https://syzygy-tables.info/ and even better for direct use in chess programs for analyzing purposes with the Fritz 17 or ChessBase 14 / 15 software.

[mbabigian]

Is it possible to schedule the download in chronological order. Chrome only accept one download at a time. My point is that once the download has finished for one particular table, the next start immediately, without any time wasted for nothing.

I don't know if it possible to download one from Sesse website and one from Lichess mirror concurrently ?

The only benefit of downloading from two different sites simultaneously would be in the case that your internet connection is faster than the path to the fastest repository. Otherwise you need two internet pipes, one for each download (i.e. home and office computers both downloading). If you have a fast internet connection >50Mb/s, pulling from multiple sites would be faster.

The tool I posted does download one after another, with the only delay being the time to hash the prior file. The cumulative hashing time is probably less than 1/60th of the total time to acquire the files. You don't want to use them without checking the file integrity.

You could cut the list of files into two text files and launch the tool twice, once for each site. So the short answer is: Yes, limited only by your creative thinking.

[syzygy]

This article from the Lomonosov tablebase page estimates that the size of an 8-man tablebase will be ~100x larger than a 7-man table base and will take a minimum of 10,000 TB of disk space and 50 TB of RAM, limiting its completion to a top-10 super computer

Syzygy 7-piece was about 1 Tb of Ram and 18Tb hard disk

Lets say you need 100 times as much ram and hard drive for each extra piece so 9 pieces needs 10 pb Ram and 180 Pb hard drive.

64 TB of RAM (shared by all cpu cores) would be enough for the current generator (with some minor tweaking).

Around 2,000 TB of disk space would probably be enough to store the full 8-piece set (WDL+DTZ).

This seems to be within the capability of Summit supercomputer

Supercomputers typically are clusters of computing nodes, with each computing node having direct access to only a small portion of total RAM memory. The current generator cannot work with that. It was designed to generate 6-piece TBs on a single machine with at least 16GB of RAM and preferably a bit more (which would have been a ridiculous requirement when Nalimov wrote his generator, but was quite reasonable in 2012). It would need 64 TB of RAM shared by all cores to generate 8-piece TBs. To tackle 8 piece, it is probably best to first write a new generator designed for the hardware that one may hope to be available 5-10 years from now.

[duncan]

This article from the Lomonosov tablebase page estimates that the size of an 8-man tablebase will be ~100x larger than a 7-man table base and will take a minimum of 10,000 TB of disk space and 50 TB of RAM, limiting its completion to a top-10 super computer

Syzygy 7-piece was about 1 Tb of Ram and 18Tb hard disk

Lets say you need 100 times as much ram and hard drive for each extra piece so 9 pieces needs 10 pb Ram and 180 Pb hard drive.

64 TB of RAM (shared by all cpu cores) would be enough for the current generator (with some minor tweaking).

Around 2,000 TB of disk space would probably be enough to store the full 8-piece set (WDL+DTZ).

This seems to be within the capability of Summit supercomputer

Supercomputers typically are clusters of computing nodes, with each computing node having direct access to only a small portion of total RAM memory. The current generator cannot work with that. It was designed to generate 6-piece TBs on a single machine with at least 16GB of RAM and preferably a bit more (which would have been a ridiculous requirement when Nalimov wrote his generator, but was quite reasonable in 2012). It would need 64 TB of RAM shared by all cores to generate 8-piece TBs. To tackle 8 piece, it is probably best to first write a new generator designed for the hardware that one may hope to be available 5-10 years from now.

Thanks for this.

HPE Integrity MC990 X Server has a maximum of 48 Tb. If this is upgraded in the future to 64 Tb, would it run your software for 8 pieces ?


https://buy.hpe.com/us/en/servers/missi ... 1008798952



Do you need a high performance server with greater in-memory computing capacity, reliability, and cost-efficiency? The HPE Integrity MC990 X Server delivers in-memory computing performance for Linux®-based applications at unparalleled scale with mission-critical reliability and modular flexibility. An advanced symmetric multiprocessing (SMP) system designed for data-intensive workloads, the HPE MC990 X Server features the enterprise-class Intel® Xeon® E7-8800/4800 v4 processors and robust reliability, availability, and serviceability. The 5U modular chassis contains 4 sockets with up to 192 threads. By adding chassis and leveraging high bandwidth, low latency NUMAlink technology, the HPE MC990 X Server can scale as a single system from 4 to 32 sockets and from 1 to 48 TB of cache-coherent shared memory.