7-men Syzygy attempt

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Dann Corbit
Posts: 10204
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: 7-men Syzygy attempt

Post by Dann Corbit » Fri May 18, 2018 2:12 am

Nordlandia wrote:
Thu May 17, 2018 6:02 pm
See 2k5/1p6/pP2K3/P7/8/4B3/8/8 w - - 0 0
Lomonosov says draw.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

User avatar
Nordlandia
Posts: 2477
Joined: Fri Sep 25, 2015 7:38 pm
Location: Sortland, Norway

Re: 7-men Syzygy attempt

Post by Nordlandia » Fri May 18, 2018 6:09 am

Dann Corbit wrote:
Fri May 18, 2018 2:12 am
Nordlandia wrote:
Thu May 17, 2018 6:02 pm
See 2k5/1p6/pP2K3/P7/8/4B3/8/8 w - - 0 0
Lomonosov says draw.
While stockfish evaluate it as +4. SF need that particular set for fortress detection.

syzygy
Posts: 4458
Joined: Tue Feb 28, 2012 10:56 pm

Re: 7-men Syzygy attempt

Post by syzygy » Sun May 20, 2018 12:02 am

syzygy wrote:
Sat May 05, 2018 12:31 am
Sesse wrote:My estimation was just wild extrapolation. People talked about 10 TB for WDL, and for the current generated 7-man TBs, DTZ files are about 2.6x the size of WDL.
The pawn tables will probably take the ratio down.

For 5-piece tables, the ratios DTZ/WDL are:

Code: Select all

             WDL        DTZ       DTZ/WDL
pawnless   68132288   287321600    4.217
pawnful   326710624   297447328    0.910
total     394842912   584768928    1.481
For 6-piece tables:

Code: Select all

              WDL            DTZ      DTZ/WDL
pawnless   11896768096   30359010784   2.552
pawnful    60937299376   57032537776   0.936
total      72834067472   87391548560   1.200
Going from 5 to 6, WDL size multiplied by 184.5. To keep 7-piece WDL below 10TB, the factor needs to be just below 151 (I am counting 1kb = 1024 :)). But Lomonosov DTM size went up by a factor of about 150, I think, and WDL and DTZ should do better than that.
We now know the 7-piece pawnless numbers:

Code: Select all

              WDL            DTZ       DTZ/WDL
pawnless 1156756806976  2010529270464   1.738
Going from 5 to 6, WDL pawnless size multiplied by 174.6. From 6 to 7, the factor is 97.2.

Going from 5 to 6, DTZ pawnless size multiplied by 105.7. From 6 to 7, the factor is 66.2.

Since the number of pawnful tables grows considerably faster than the number of pawnless tables, the factors for pawnful tables will be higher.

syzygy
Posts: 4458
Joined: Tue Feb 28, 2012 10:56 pm

Re: 7-men Syzygy attempt

Post by syzygy » Sun May 20, 2018 12:34 am

syzygy wrote:
Sun May 20, 2018 12:02 am
Since the number of pawnful tables grows considerably faster than the number of pawnless tables, the factors for pawnful tables will be higher.
Number of pawnless tables with 5, 6, 7 pieces: 60, 170, 396.
Average WDL sizes: 1135538, 69980989, 2921103047.
Ratio from 5 to 6 and from 6 to 7: 61.63 and 41.74.

Number of pawnful tables with 5, 6, 7 pieces: 50, 195, 605.
Average WDL sizes: 6534212, 312498971, ???
Ratio from 5 to 6: 47.83.

If a pawnful 7-piece WDL table is on average 40x the size of a pawnful 6-piece WDL table, we get 6.878 TB 7-piece pawnful WDL. Together with the 1.052 TB 7-piece pawnless WDL that results in 7.93 TB 7-piece WDL.

So 8 TB seems to be a reasonable estimate, which means about 8.8 TB in hard-disk TBs.

syzygy
Posts: 4458
Joined: Tue Feb 28, 2012 10:56 pm

Re: 7-men Syzygy attempt

Post by syzygy » Sun May 20, 2018 12:42 am

http://kirill-kryukov.com/chess/nulp/results.html

The number of legal positions with 5, 6 and 7 pieces is 25912594054, 3787154440416 and 423836835667331.

Compression ratio for 5-piece WDL is 65.6276 positions per byte.
Compression ratio for 6-piece WDL is 51.9970 positions per byte.

For 7-piece WDL to remain under 8 TB, we need 48.1847 positions per byte.

User avatar
cdani
Posts: 2104
Joined: Sat Jan 18, 2014 9:24 am
Location: Andorra
Contact:

Re: 7-men Syzygy attempt

Post by cdani » Sun May 20, 2018 5:51 am

syzygy wrote:
Sun May 20, 2018 12:42 am
http://kirill-kryukov.com/chess/nulp/results.html

The number of legal positions with 5, 6 and 7 pieces is 25912594054, 3787154440416 and 423836835667331.

Compression ratio for 5-piece WDL is 65.6276 positions per byte.
Compression ratio for 6-piece WDL is 51.9970 positions per byte.

For 7-piece WDL to remain under 8 TB, we need 48.1847 positions per byte.
In any case, an impressive work!! Congratulations!!

Modern Times
Posts: 2420
Joined: Thu Jun 07, 2012 9:02 pm

Re: 7-men Syzygy attempt

Post by Modern Times » Sun May 20, 2018 7:48 am

http://kirill-kryukov.com/chess/nulp/results.html

I found the chess960 bit fascinating, near the bottom.

I am reading this correctly - in chess960, 3.565% of 8-piece endgame positions could not have been reached in normal chess ?
.

Opinions expressed here are my own, and not necessarily those of the CCRL Group.

noobpwnftw
Posts: 362
Joined: Sun Nov 08, 2015 10:10 pm

Re: 7-men Syzygy attempt

Post by noobpwnftw » Mon May 21, 2018 2:09 pm

syzygy wrote:
Mon May 14, 2018 7:29 pm
duncan wrote:
Mon May 14, 2018 7:14 pm
do you know if the new generator will also work for 8 pieces ?
In theory it should work. In practice you'll need a very very big machine and a lot of patience.

It would probably be a good idea to rewrite the generator to use a more space-efficient indexing scheme during generation. Generation of the biggest table should then fit in about 41 TB (so let's say 48 TB) and most tables would need half of that or less. The current scheme would need 57.75 TB (so basically a 64 TB machine).

The bigger 8-piece tables will fill more than 2^32 64-byte blocks with compressed data, which means the format will need a relatively small extension to deal with them.
In practice, building full 7-piece tables is going to take more than half a year on my two machines, with 8-piece at current speed it probably gonna take a century to build the full set. So I'd just wait for the hardware developments to reach that reasonable level first.

While we wait, I'm getting extra storage ready for other interesting metrics(like DTM50) or variations(antichess, suiside) if there is interest, so after having full 7-piece DTZ50 on standard done, shall we try building stuff like those?

syzygy
Posts: 4458
Joined: Tue Feb 28, 2012 10:56 pm

Re: 7-men Syzygy attempt

Post by syzygy » Mon May 21, 2018 4:32 pm

noobpwnftw wrote:
Mon May 21, 2018 2:09 pm
In practice, building full 7-piece tables is going to take more than half a year on my two machines, with 8-piece at current speed it probably gonna take a century to build the full set. So I'd just wait for the hardware developments to reach that reasonable level first.
It could be interesting to generate a few well-chosen 8-piece tables to try to beat the current max DTZ record of 517 moves. That should be possible on one of your machines, but it would need a new generator. (And it is not clear that the max 8-piece DTZ exceeds the max 7-piece DTZ.)
While we wait, I'm getting extra storage ready for other interesting metrics(like DTM50) or variations(antichess, suiside) if there is interest, so after having full 7-piece DTZ50 on standard done, shall we try building stuff like those?
The DTM50 value of a position varies with the value of the 50-move counter, so that metric is rather complicated both for generation and for storage. At the moment I don't know if this is doable.

Regular DTM tables would replicate the Lomonosov tables. I have a mostly finished DTM generator, but it needs too much memory now for pawns (it generates the whole table at once instead of by file, which was easier to implement since I compress these tables by rank). There may also be some performance problems that need to be addressed somehow. Total 7-piece DTM size would be of the order of the Lomonosov TBs (storing only the "smaller half" of them, like I do for DTZ tables, is not a real option since information for both sides is needed to handle promotions).

The problem of antichess/suicide chess is the sheer number of tables. There are 3570 pawnless 6-piece antichess tables and 4830 pawnful ones. I have not yet calculated the number of 7-piece antichess tables. The existing 6-piece tables at the lichess.ovh server have extra compression (essentially eliminating positions with a winning threat, i.e. a move that forces a losing capture by the other side), which would probably slow down generation of 7-piece tables too much. So those would need to be regenerated (approximately doubling their size, I think).

Atomic chess DTZ tables are probably the most similar to normal chess DTZ tables. The main complication is that kings can connect on the board in atomic chess (and also in antichess/suicide chess), which means that there are more than 462 king-king configurations. So the pawnless generator needs more memory.

noobpwnftw
Posts: 362
Joined: Sun Nov 08, 2015 10:10 pm

Re: 7-men Syzygy attempt

Post by noobpwnftw » Mon May 21, 2018 4:55 pm

syzygy wrote:
Mon May 21, 2018 4:32 pm
noobpwnftw wrote:
Mon May 21, 2018 2:09 pm
In practice, building full 7-piece tables is going to take more than half a year on my two machines, with 8-piece at current speed it probably gonna take a century to build the full set. So I'd just wait for the hardware developments to reach that reasonable level first.
It could be interesting to generate a few well-chosen 8-piece tables to try to beat the current max DTZ record of 517 moves. That should be possible on one of your machines, but it would need a new generator. (And it is not clear that the max 8-piece DTZ exceeds the max 7-piece DTZ.)
While we wait, I'm getting extra storage ready for other interesting metrics(like DTM50) or variations(antichess, suiside) if there is interest, so after having full 7-piece DTZ50 on standard done, shall we try building stuff like those?
The DTM50 value of a position varies with the value of the 50-move counter, so that metric is rather complicated both for generation and for storage. At the moment I don't know if this is doable.

Regular DTM tables would replicate the Lomonosov tables. I have a mostly finished DTM generator, but it needs too much memory now for pawns (it generates the whole table at once instead of by file, which was easier to implement since I compress these tables by rank). There may also be some performance problems that need to be addressed somehow. Total 7-piece DTM size would be of the order of the Lomonosov TBs (storing only the "smaller half" of them, like I do for DTZ tables, is not a real option since information for both sides is needed to handle promotions).

The problem of antichess/suicide chess is the sheer number of tables. There are 3570 pawnless 6-piece antichess tables and 4830 pawnful ones. I have not yet calculated the number of 7-piece antichess tables. The existing 6-piece tables at the lichess.ovh server have extra compression (essentially eliminating positions with a winning threat, i.e. a move that forces a losing capture by the other side), which would probably slow down generation of 7-piece tables too much. So those would need to be regenerated (approximately doubling their size, I think).

Atomic chess DTZ tables are probably the most similar to normal chess DTZ tables. The main complication is that kings can connect on the board in atomic chess (and also in antichess/suicide chess), which means that there are more than 462 king-king configurations. So the pawnless generator needs more memory.
Thanks for all the information, I think you have just answered a lot of potential questions that may come forward :).
I guess one could now look at the obstacles and efforts needed to move forward.

Post Reply