Syzygy DTZ data explaination?

[phhnguyen]

I used to work with Syzygy WDL and everything is fine. The results of W/D/L are easy to understand. However, when looking into DTZ data I am so confused since the data is not self-explanation to me. The result only (winning line) is not enough for me since I have been working on a studying tool to work with Syzygy. Below is the output of Fathom for a position KRvsK. I focused on moves which are returned from DTZ probe.

Questions:

- What does the rank 1000 mean?
- What does the score 31744 mean?
- Why all moves have the same rank (1000) and score (31744), both DTZ and WDL? What extra information DTZ has added, compared with WDL?
- The winning line starts with move Rc2, how to get/compute that 1st move (Rc2)?
- There is another winning line starts with Rg6 and it is as good as Rc2's one. Is there any way to find it out?

Thanks

Code: Select all

[Event ""]
[Site ""]
[Date "??"]
[Round "-"]
[White "Syzygy"]
[Black "Syzygy"]
[Result "1-0"]
[FEN "K7/8/2R5/8/8/8/8/7k w - - 0 1"]
[WDL "Win"]
[DTZ "21"]
[WinningMoves "Ka7, Kb7, Kb8, Rc1, Rc2, Rc3, Rc4, Rc5, Ra6, Rb6, Rd6, Re6, Rf6, Rg6, Rh6, Rc7, Rc8"]
[DrawingMoves ""]
[LosingMoves ""]

1. Rc2 Kg1 2. Ka7 Kf1 3. Kb6 Ke1 4. Kc5 Kd1 5. Rh2 Kc1 6. Kd4 Kd1 7. Ke3 Kc1 8. Kd3 Kb1 9. Kc3 Ka1 10. Kb3 Kb1 11. Rh1# 1-0
17 moves returned from DTZ probe
Ka7 rank = 1000 score=31744
Kb7 rank = 1000 score=31744
Kb8 rank = 1000 score=31744
Rc1 rank = 1000 score=31744
Rc2 rank = 1000 score=31744
Rc3 rank = 1000 score=31744
Rc4 rank = 1000 score=31744
Rc5 rank = 1000 score=31744
Ra6 rank = 1000 score=31744
Rb6 rank = 1000 score=31744
Rd6 rank = 1000 score=31744
Re6 rank = 1000 score=31744
Rf6 rank = 1000 score=31744
Rg6 rank = 1000 score=31744
Rh6 rank = 1000 score=31744
Rc7 rank = 1000 score=31744
Rc8 rank = 1000 score=31744
17 moves returned from WDL probe
Ka7 rank = 1000 score=31744
Kb7 rank = 1000 score=31744
Kb8 rank = 1000 score=31744
Rc1 rank = 1000 score=31744
Rc2 rank = 1000 score=31744
Rc3 rank = 1000 score=31744
Rc4 rank = 1000 score=31744
Rc5 rank = 1000 score=31744
Ra6 rank = 1000 score=31744
Rb6 rank = 1000 score=31744
Rd6 rank = 1000 score=31744
Re6 rank = 1000 score=31744
Rf6 rank = 1000 score=31744
Rg6 rank = 1000 score=31744
Rh6 rank = 1000 score=31744
Rc7 rank = 1000 score=31744
Rc8 rank = 1000 score=31744

The position:
[d]K7/8/2R5/8/8/8/8/7k w - - 0 1

[Gian-Carlo Pascutto]

Fathom ranks (and scores) all winning moves the same unless the probe code indicates the position has been repeated (i.e. that the engine is failing to make progress by itself).

[Joerg Oster]

I guess you want to rank the moves strictly by their dtz values.
This is what I do here https://github.com/joergoster/Stockfish ... 3181ef95fb
in a stockfish branch to build a mating sequence.

This is what I get with your position:

Code: Select all

position fen K7/8/2R5/8/8/8/8/7k w - - 0 1
go
info depth 0 seldepth 21 multipv 1 score mate 11 nodes 685 nps 114166 tbhits 235 time 6 pv c6c2 h1g1 a8b7 g1f1 b7c6 f1e1 c6d5 e1d1 c2h2 d1c1 d5e4 c1b1 h2e2 b1a1 e4d3 a1b1 d3c3 b1a1 c3b3 a1b1 e2e1
bestmove c6c2 ponder h1g1

There are some more examples in the comment at the bottom.

[phhnguyen]

Fathom ranks (and scores) all winning moves the same unless the probe code indicates the position has been repeated (i.e. that the engine is failing to make progress by itself).

Thanks!

After telling the position is repeated, the data looks more reasonable. It could explain why Rc2 and Rg6 (by ranks)!

I guess the rank is the distance to mate/convert. However, the score is still a mystery for me.

Code: Select all

[Event ""]
[Site ""]
[Date "??"]
[Round "-"]
[White "Syzygy"]
[Black "Syzygy"]
[Result "1-0"]
[FEN "K7/8/2R5/8/8/8/8/7k w - - 0 1"]
[WDL "Win"]
[DTZ "21"]
[WinningMoves "Ka7, Kb7, Kb8, Rc1, Rc2, Rc3, Rc4, Rc5, Ra6, Rb6, Rd6, Re6, Rf6, Rg6, Rh6, Rc7, Rc8"]
[DrawingMoves ""]
[LosingMoves ""]

1. Rc2 Kg1 2. Ka7 Kf1 3. Kb6 Ke1 4. Kc5 Kd1 5. Rh2 Kc1 6. Kd4 Kd1 7. Ke3 Kc1 8. Kd3 Kb1 9. Kc3 Ka1 10. Kb3 Kb1 11. Rh1# 1-0
17 moves returned from DTZ probe
Ka7 rank = 973 score=31744
Kb7 rank = 973 score=31744
Kb8 rank = 973 score=31744
Rc1 rank = 971 score=31744
Rc2 rank = 979 score=31744
Rc3 rank = 973 score=31744
Rc4 rank = 971 score=31744
Rc5 rank = 971 score=31744
Ra6 rank = 973 score=31744
Rb6 rank = 971 score=31744
Rd6 rank = 971 score=31744
Re6 rank = 971 score=31744
Rf6 rank = 973 score=31744
Rg6 rank = 979 score=31744
Rh6 rank = 971 score=31744
Rc7 rank = 971 score=31744
Rc8 rank = 973 score=31744
17 moves returned from WDL probe
Ka7 rank = 1000 score=31744
Kb7 rank = 1000 score=31744
Kb8 rank = 1000 score=31744
Rc1 rank = 1000 score=31744
Rc2 rank = 1000 score=31744
Rc3 rank = 1000 score=31744
Rc4 rank = 1000 score=31744
Rc5 rank = 1000 score=31744
Ra6 rank = 1000 score=31744
Rb6 rank = 1000 score=31744
Rd6 rank = 1000 score=31744
Re6 rank = 1000 score=31744
Rf6 rank = 1000 score=31744
Rg6 rank = 1000 score=31744
Rh6 rank = 1000 score=31744
Rc7 rank = 1000 score=31744
Rc8 rank = 1000 score=31744

[phhnguyen]

I guess you want to rank the moves strictly by their dtz values.

Almost! My purpose is to explain for a given endgame why this move should be made but not others.

I lost when following your code for a while. Do you get information to rank all moves at once (query one) or do you make-query one by one?

(actually, I am not sure what Fathom code does, but I just call that code one for ranking all moves of a given position)

[Joerg Oster]

I guess you want to rank the moves strictly by their dtz values.

Almost! My purpose is to explain for a given endgame why this move should be made but not others.

I lost when following your code for a while. Do you get information to rank all moves at once (query one) or do you make-query one by one?

(actually, I am not sure what Fathom code does, but I just call that code one for ranking all moves of a given position)

One query gives you information for all legal moves in that position.

As I try to build the full mating sequence, I have to query every time a move is made until the mate is reached.

[phhnguyen]

The problem has not been solved yet as I thought: Indicating repetition to Fathon code does not always give me enough information.
For losing positions all ranks are the same again:

Code: Select all

[Event ""]
[Site ""]
[Date "??"]
[Round "-"]
[White "Syzygy"]
[Black "Syzygy"]
[Result "1-0"]
[FEN "K7/8/8/8/8/2R5/8/7k b - - 0 1"]
[WDL "Loss"]
[DTZ "26"]
[WinningMoves ""]
[DrawingMoves ""]
[LosingMoves "Kg1, Kg2, Kh2"]

1... Kg2 2. Ka7 Kf2 3. Ka6 Ke2 4. Kb5 Kd1 5. Kc4 Kd2 6. Kb3 Kd1 7. Rc2 Ke1 8. Kb2 Kd1 9. Kc3 Ke1 10. Kd3 Kf1 11. Ke3 Kg1 12. Kf3 Kh1 13. Kg3 Kg1 14. Rc1# 1-0
3 moves returned from DTZ probe
Kg1 rank = -1000 score=-31744
Kg2 rank = -1000 score=-31744
Kh2 rank = -1000 score=-31744
3 moves returned from WDL probe
Kg1 rank = -1000 score=-31744
Kg2 rank = -1000 score=-31744
Kh2 rank = -1000 score=-31744
Program ended with exit code: 0

[d]K7/8/8/8/8/2R5/8/7k b - - 0 1

[syzygy]

The problem has not been solved yet as I thought: Indicating repetition to Fathon code does not always give me enough information.

I don't know what Fathom does, but in Stockfish ranks work as follows (it may be the same for Fathom).

If the current position is in the TBs, each move is assigned a rank indicating how "good" the move is that is independent of what other moves may be available (which makes the ranking system work with UCI searchmoves and multipv).

Rank 1000 is a move that is certain to win while respecting the 50-move rule.

Rank 1-900 are moves that lead to a "cursed" win, i.e. the opponent can draw by the 50-move rule. Rank 900 misses the win by a hair (so will in fact be won if the opponent plays one inaccurate move), rank 1 misses the win by a whole lot.

(Rank 901-999 is possible if there has been a single repetition but the game can still be reliably won by playing DTZ-optimal moves.)

Rank 0 are draws.

Rank -1000 are losses with little chance to reach the 50-move rule.

Rank -999 to -1 are losses with some drawing chances (certain to draw starting from -900).

To be precise:

Code: Select all

// Better moves are ranked higher. Certain wins are ranked equally.
// Losing moves are ranked equally unless a 50-move draw is in sight.
int r =  dtz > 0 ? (dtz + cnt50 <= 99 && !rep ? 1000 : 1000 - (dtz + cnt50))
       : dtz < 0 ? (-dtz * 2 + cnt50 < 100 ? -1000 : -1000 + (-dtz + cnt50))
       : 0;

NOTE: rep == true if there has been a repetition since the last zeroing move. It does not need to be the current root position that has repeated. I don't know if Fathom handles this correctly. I'm guessing it does not, but I have not checked.

With this ranking system, the engine can avoid losing any full or half points once it has reached a TB position by making sure that it always plays one of the highest-ranked moves. This is the sole purpose of the ranks. They are not intended to be displayed to the user.

[phhnguyen]

Thanks a lot for the very detail and helpful explanation!

Can you help me confirm what I have just guessed for the above case (position K7/8/8/8/8/2R5/8/7k b - - 0 1)
1) All raw data DTZ of all three black-King moves (Kg1, Kg2, Kh2) has the same rank -1000 (losses with little chance to reach the 50-move rule)
2) If all moves (of a given position) have the same rank (e.g. -1000) we need to search one ply to find progress

With this ranking system, the engine can avoid losing any full or half points once it has reached a TB position by making sure that it always plays one of the highest-ranked moves. This is the sole purpose of the ranks. They are not intended to be displayed to the user.

IMHO, any raw data such as your ranking system have a special beauty and usefulness, at least for computer chess programmers. They can answer many questions we have about using Syzygy Egtb.

The only thing I concern about is that from them (ranks) only I may not find out how to progress/pick up best moves for some positions.

[syzygy]

Can you help me confirm what I have just guessed for the above case (position K7/8/8/8/8/2R5/8/7k b - - 0 1)
1) All raw data DTZ of all three black-King moves (Kg1, Kg2, Kh2) has the same rank -1000 (losses with little chance to reach the 50-move rule)

Your position has three moves, all losing with DTZ=25, 23 and 23 (ply).
https://syzygy-tables.info/?fen=K7/8/8/ ... _b_-_-_0_1

With the 50-move counter at 0, there are 100 ply left so there is little chance of reaching the 50-move rule. So Stockfish internally ranks them equally (all -1000). Since they are all ranked equally, Stockfish searches all three of them and plays the move that it thinks is best (= least bad). Therefore SF might choose to play Kg1 with DTZ=23 even though Kg2 has DTZ=25.

2) If all moves (of a given position) have the same rank (e.g. -1000) we need to search one ply to find progress

In this case everything loses, so the engine would prefer not to make any progress at all.

As long as the engine plays moves with rank 1000, it is guaranteed to win. Stockfish performs a search on the moves with rank 1000 (ignoring all lower-ranked moves) and plays the move with the best search score. This move will almost always "look good".

The only thing I concern about is that from them (ranks) only I may not find out how to progress/pick up best moves for some positions.

If you are only interested in making progress in the sense of minimising DTZ (if the engine is winning), then you can just let the engine play a winning move that resets the 50-move rule or, if none is available, the winning move that minimised DTZ.

However, this results in ugly play, e.g. in this position:
https://syzygy-tables.info/?fen=K7/8/8/ ... _b_-_-_0_1
Kxh2 resets the 50-move counter, so is the "worst possible" move. The losing engine would play Kg2 instead (DTZ=15). (Note that Kxh2 would in fact be the only losing move if the 50-move counter were almost depleted!)

Or this position:
https://syzygy-tables.info/?fen=K7/2Q5/ ... _w_-_-_0_1
The DTZ-minimising engine would play Qh2+, which indeed minimises DTZ as it forces the losing side to capture the Q. (Note that if the 50-move counter were at 49 moves, then Qh2+ would be the only winning move! In that case, Qh2+ would be ranked 1000 and the other moves 900 and below.)

To avoid such unnatural moves (that do not "look good"), it is better to let the engine do a normal search on the subset of positions that are known to preserve the game's outcome.