How is work on 8-man tablebases progressing?

[Joerg Oster]

I think 8-man can give 0-5 elo gain like 7-man. No practical benefit. Only academic interest. I don't use even 6-man. If there is interesting position I check DTM from Shredderchess .

You are wrong.
With current technology 8 man will lose elo because they will slow down engine a lot during probing because of their enormous size.

No.
It is not going to lose elo because engines are not going to use it if it lose elo.
I see no reason to use it in every node and engines can use it only in the first 1% of the time that they search.

Engines will use it or not use it solely dependent on how they are configured by people who run them.
I've myself seen stockfish having 1,5x slowdown in midgame because of slow 7 men TBs, for example, and person who ran it said it's "stockfish bug" and not him putting TBs on really slow SSD.
As per usual you confidently write smth in area which you have literally 0 clue about.

I assumed logical decision by programmers in order to win games.
For the claim it is a stockfish bug I will not say bug because it is something that the programmers did on purpose but I wonder why stockfish does not have a code to probe the tablebases less often if it has a significant slowdown when it start probing.

There is an UCI option which allows the user to control the TB probing rate to some extent.

SyzygyProbeDepth type spin default 1 min 1 max 100
Minimum remaining search depth for which a position is probed. Set this option to a higher value to probe less aggressively if you experience too much slowdown (in terms of nps) due to tablebase probing.

[syzygy]

There is an UCI option which allows the user to control the TB probing rate to some extent.

SyzygyProbeDepth type spin default 1 min 1 max 100
Minimum remaining search depth for which a position is probed. Set this option to a higher value to probe less aggressively if you experience too much slowdown (in terms of nps) due to tablebase probing.

Indeed. So setting it to 100 will effectively limit probing to the root node.

Implementing other ways of limiting the probing rate is an insignificant amount of work, certainly compared with generating the 8-men tables.

[hgm]

You argue like completion of the tables is a goal in itself. But the goal is to improve analysis of chess positions. And who would care if only 90% of the tables are available, if these would cover 99.99% of all positions people want to probe? Completing the table would be an effort that no one cares about.

[noobpwnftw]

"If there's one Jedi left, it's not you."

[syzygy]

You argue like completion of the tables is a goal in itself.

Correct.

Still, I am looking forward to the partial 8-men tables you will be generating in the coming months.

[Koistinen]

Summary:
There is no progress into 8-man tablebases and a major stumbling block is the storage requirement.
A distributed solution would use about 2 EB which might be reduced to 100 PB by symmetry and then perhaps to 20 PB by compression.
Cost of storage would then become about 1000 * 20 TB disks which would be about 400 kUSD just for the disks.

The usefulness of this is in doubt as the 6 and 7-man tablebases don't see much use.
Use by chess engines could be improved.
Further compression looks like a fertile field.
The intended software improvements could also be applied to 6 and 7-man.

The main use case for the 7-man seems to be postal chess where they have become part of the game.

In conclusion, it is better to first improve 7-man before going for 8-man.

[syzygy]

Summary:
There is no progress into 8-man tablebases and a major stumbling block is the storage requirement.
A distributed solution would use about 2 EB which might be reduced to 100 PB by symmetry and then perhaps to 20 PB by compression.
Cost of storage would then become about 1000 * 20 TB disks which would be about 400 kUSD just for the disks.

The usefulness of this is in doubt as the 6 and 7-man tablebases don't see much use.
Use by chess engines could be improved.
Further compression looks like a fertile field.
The intended software improvements could also be applied to 6 and 7-man.

The main use case for the 7-man seems to be postal chess where they have become part of the game.

In conclusion, it is better to first improve 7-man before going for 8-man.

What one could do is generate 8-men (or even 7-men) tables for positions with blocked pawns. Such tables would probably be quite useful.

I think the "Robbobases" covered such positions. They basically treated a pair of blocked pawns as one piece. So one could have 9-men tables with 2 pairs of blocked pawns alongside 7-men tables, and even 10-men tables with 3 pairs of blocked pawns and 11-men tables with 4 pairs of blocked pawns (plus 2 kings plus one further piece, e.g. KNPPPPvKPPPP).

[phhnguyen]

In conclusion, it is better to first improve 7-man before going for 8-man.

What do you expect to be improvements for 7-man?

[Uri Blass]

In conclusion, it is better to first improve 7-man before going for 8-man.

What do you expect to be improvements for 7-man?

Do we have distance to mate in 7 piece tablebases without the fifty move rule and not only distance to conversion?
If we do not then finding the distance to mate is an improvement.

Edit:I see that there is a claim that it is 549 moves

https://chess.stackexchange.com/questio ... -checkmate

The problem is that I remember reading that lomonosov lost his tablebases and I am not sure if we have access for them.

[syzygy]

In conclusion, it is better to first improve 7-man before going for 8-man.

What do you expect to be improvements for 7-man?

Do we have distance to mate in 7 piece tablebases without the fifty move rule and not only distance to conversion?
If we do not then finding the distance to mate is an improvement.

The "Lomonosov" 7-piece DTM tables exist, although I don't think they are available for download. They are reported to be 140 TB.

But are they more usable than the existing 7-piece tables? At least I think Urban is thinking of making the tables "more usable", which I suppose means smaller without making them too slow.

In principle all tables can be compressed down to the tiny size of the generator that generates them. But the result would be unusably slow. So there is a trade-off to be made between size and access speed (and usability of the information itself, e.g. DTZ vs DTM, 50-move info, etc.).

I think the Syzygy WDL/DTZ tables strike a pretty good balance (but preferring speed over size) that will not be easy to beat by a significant margin.

I have also generated DTM tables, but I could not decide whether they should be 2-sided or 1-sided. You'd probably want to probe them during the search once a TB win has been identified, which suggests 2-sided would be preferred, but those get pretty big (on the other hand, what's another terabyte of data today). Unfortunately Stockfish(/Cfish) wasn't very good at resolving the DTM value after a TB win had been found.

Btw, a DTM table will almost necessarily have to ignore the 50-move rule because the DTM value of a position would otherwise be dependent on the value of the 50-move counter. So you'd have to store 50x as much information (in practice this would compress pretty well, but still you end up with a lot more data, and the generation phase also needs more resources). Some people have worked on "DTM50" tables, but I doubt they have gone beyond 5-piece tables.

The problem is that I remember reading that lomonosov lost his tablebases and I am not sure if we have access for them.

I think I read somewhere that they are no longer accessible through the website, but with 140 TB of data I guess there is a chance that a part of all of it was lost. Maybe the HDDs were scrapped for parts...