7-men Syzygy attempt

[Dann Corbit]

See 2k5/1p6/pP2K3/P7/8/4B3/8/8 w - - 0 0

Lomonosov says draw.

[Nordlandia]

See 2k5/1p6/pP2K3/P7/8/4B3/8/8 w - - 0 0

Lomonosov says draw.

While stockfish evaluate it as +4. SF need that particular set for fortress detection.

[syzygy]

My estimation was just wild extrapolation. People talked about 10 TB for WDL, and for the current generated 7-man TBs, DTZ files are about 2.6x the size of WDL.

The pawn tables will probably take the ratio down.

For 5-piece tables, the ratios DTZ/WDL are:

Code: Select all

             WDL        DTZ       DTZ/WDL
pawnless   68132288   287321600    4.217
pawnful   326710624   297447328    0.910
total     394842912   584768928    1.481

For 6-piece tables:

Code: Select all

              WDL            DTZ      DTZ/WDL
pawnless   11896768096   30359010784   2.552
pawnful    60937299376   57032537776   0.936
total      72834067472   87391548560   1.200

Going from 5 to 6, WDL size multiplied by 184.5. To keep 7-piece WDL below 10TB, the factor needs to be just below 151 (I am counting 1kb = 1024 ). But Lomonosov DTM size went up by a factor of about 150, I think, and WDL and DTZ should do better than that.

We now know the 7-piece pawnless numbers:

Code: Select all

              WDL            DTZ       DTZ/WDL
pawnless 1156756806976  2010529270464   1.738

Going from 5 to 6, WDL pawnless size multiplied by 174.6. From 6 to 7, the factor is 97.2.

Going from 5 to 6, DTZ pawnless size multiplied by 105.7. From 6 to 7, the factor is 66.2.

Since the number of pawnful tables grows considerably faster than the number of pawnless tables, the factors for pawnful tables will be higher.

[syzygy]

Since the number of pawnful tables grows considerably faster than the number of pawnless tables, the factors for pawnful tables will be higher.

Number of pawnless tables with 5, 6, 7 pieces: 60, 170, 396.
Average WDL sizes: 1135538, 69980989, 2921103047.
Ratio from 5 to 6 and from 6 to 7: 61.63 and 41.74.

Number of pawnful tables with 5, 6, 7 pieces: 50, 195, 605.
Average WDL sizes: 6534212, 312498971, ???
Ratio from 5 to 6: 47.83.

If a pawnful 7-piece WDL table is on average 40x the size of a pawnful 6-piece WDL table, we get 6.878 TB 7-piece pawnful WDL. Together with the 1.052 TB 7-piece pawnless WDL that results in 7.93 TB 7-piece WDL.

So 8 TB seems to be a reasonable estimate, which means about 8.8 TB in hard-disk TBs.

[syzygy]

http://kirill-kryukov.com/chess/nulp/results.html

The number of legal positions with 5, 6 and 7 pieces is 25912594054, 3787154440416 and 423836835667331.

Compression ratio for 5-piece WDL is 65.6276 positions per byte.
Compression ratio for 6-piece WDL is 51.9970 positions per byte.

For 7-piece WDL to remain under 8 TB, we need 48.1847 positions per byte.

[cdani]

http://kirill-kryukov.com/chess/nulp/results.html

The number of legal positions with 5, 6 and 7 pieces is 25912594054, 3787154440416 and 423836835667331.

Compression ratio for 5-piece WDL is 65.6276 positions per byte.
Compression ratio for 6-piece WDL is 51.9970 positions per byte.

For 7-piece WDL to remain under 8 TB, we need 48.1847 positions per byte.

In any case, an impressive work!! Congratulations!!

[Modern Times]

http://kirill-kryukov.com/chess/nulp/results.html

I found the chess960 bit fascinating, near the bottom.

I am reading this correctly - in chess960, 3.565% of 8-piece endgame positions could not have been reached in normal chess ?

[noobpwnftw]

do you know if the new generator will also work for 8 pieces ?

In theory it should work. In practice you'll need a very very big machine and a lot of patience.

It would probably be a good idea to rewrite the generator to use a more space-efficient indexing scheme during generation. Generation of the biggest table should then fit in about 41 TB (so let's say 48 TB) and most tables would need half of that or less. The current scheme would need 57.75 TB (so basically a 64 TB machine).

The bigger 8-piece tables will fill more than 2^32 64-byte blocks with compressed data, which means the format will need a relatively small extension to deal with them.

In practice, building full 7-piece tables is going to take more than half a year on my two machines, with 8-piece at current speed it probably gonna take a century to build the full set. So I'd just wait for the hardware developments to reach that reasonable level first.

While we wait, I'm getting extra storage ready for other interesting metrics(like DTM50) or variations(antichess, suiside) if there is interest, so after having full 7-piece DTZ50 on standard done, shall we try building stuff like those?

[syzygy]

In practice, building full 7-piece tables is going to take more than half a year on my two machines, with 8-piece at current speed it probably gonna take a century to build the full set. So I'd just wait for the hardware developments to reach that reasonable level first.

It could be interesting to generate a few well-chosen 8-piece tables to try to beat the current max DTZ record of 517 moves. That should be possible on one of your machines, but it would need a new generator. (And it is not clear that the max 8-piece DTZ exceeds the max 7-piece DTZ.)

While we wait, I'm getting extra storage ready for other interesting metrics(like DTM50) or variations(antichess, suiside) if there is interest, so after having full 7-piece DTZ50 on standard done, shall we try building stuff like those?

The DTM50 value of a position varies with the value of the 50-move counter, so that metric is rather complicated both for generation and for storage. At the moment I don't know if this is doable.

Regular DTM tables would replicate the Lomonosov tables. I have a mostly finished DTM generator, but it needs too much memory now for pawns (it generates the whole table at once instead of by file, which was easier to implement since I compress these tables by rank). There may also be some performance problems that need to be addressed somehow. Total 7-piece DTM size would be of the order of the Lomonosov TBs (storing only the "smaller half" of them, like I do for DTZ tables, is not a real option since information for both sides is needed to handle promotions).

The problem of antichess/suicide chess is the sheer number of tables. There are 3570 pawnless 6-piece antichess tables and 4830 pawnful ones. I have not yet calculated the number of 7-piece antichess tables. The existing 6-piece tables at the lichess.ovh server have extra compression (essentially eliminating positions with a winning threat, i.e. a move that forces a losing capture by the other side), which would probably slow down generation of 7-piece tables too much. So those would need to be regenerated (approximately doubling their size, I think).

Atomic chess DTZ tables are probably the most similar to normal chess DTZ tables. The main complication is that kings can connect on the board in atomic chess (and also in antichess/suicide chess), which means that there are more than 462 king-king configurations. So the pawnless generator needs more memory.

[noobpwnftw]

In practice, building full 7-piece tables is going to take more than half a year on my two machines, with 8-piece at current speed it probably gonna take a century to build the full set. So I'd just wait for the hardware developments to reach that reasonable level first.

It could be interesting to generate a few well-chosen 8-piece tables to try to beat the current max DTZ record of 517 moves. That should be possible on one of your machines, but it would need a new generator. (And it is not clear that the max 8-piece DTZ exceeds the max 7-piece DTZ.)

While we wait, I'm getting extra storage ready for other interesting metrics(like DTM50) or variations(antichess, suiside) if there is interest, so after having full 7-piece DTZ50 on standard done, shall we try building stuff like those?

The DTM50 value of a position varies with the value of the 50-move counter, so that metric is rather complicated both for generation and for storage. At the moment I don't know if this is doable.

Regular DTM tables would replicate the Lomonosov tables. I have a mostly finished DTM generator, but it needs too much memory now for pawns (it generates the whole table at once instead of by file, which was easier to implement since I compress these tables by rank). There may also be some performance problems that need to be addressed somehow. Total 7-piece DTM size would be of the order of the Lomonosov TBs (storing only the "smaller half" of them, like I do for DTZ tables, is not a real option since information for both sides is needed to handle promotions).

The problem of antichess/suicide chess is the sheer number of tables. There are 3570 pawnless 6-piece antichess tables and 4830 pawnful ones. I have not yet calculated the number of 7-piece antichess tables. The existing 6-piece tables at the lichess.ovh server have extra compression (essentially eliminating positions with a winning threat, i.e. a move that forces a losing capture by the other side), which would probably slow down generation of 7-piece tables too much. So those would need to be regenerated (approximately doubling their size, I think).

Atomic chess DTZ tables are probably the most similar to normal chess DTZ tables. The main complication is that kings can connect on the board in atomic chess (and also in antichess/suicide chess), which means that there are more than 462 king-king configurations. So the pawnless generator needs more memory.

Thanks for all the information, I think you have just answered a lot of potential questions that may come forward .
I guess one could now look at the obstacles and efforts needed to move forward.