7-men Syzygy attempt

[syzygy]

The WDL problems now disappear, so KRBNvKQN.rtbw seems to be correct.
There are still problems with DTZ/KRBNvKQN.rtbz:

Fen: 5q1n/8/8/4N3/4B3/3K2R1/8/4k3 w - - 0 1
Key: 8AF65EE2CD3390E8
Checkers:
Tablebases WDL: 2 (1)
Tablebases DTZ: 10 (1)
Tablebases DTM: cp 0 (0)
(Should be DTZ 65)

Fen: 5n2/8/8/7q/3N1R2/3B4/8/k2K4 w - - 0 1
Key: 99AC33FADD29C35A
Checkers: h5
Tablebases WDL: 1 (1)
Tablebases DTZ: 110 (2)
Tablebases DTM: cp 0 (0)
(Should be DTZ 152 +/- 1)

Fen: 8/2B5/8/1k6/8/3K4/4N1R1/qn6 w - - 0 1
Key: 87683D8F8EA45EB4
Checkers:
Tablebases WDL: -1 (1)
Tablebases DTZ: -454 (1)
Tablebases DTM: mate 0 (0)
(Should be DTZ -1034 +/- 1)

Fen: 8/8/8/q7/8/8/1R3B2/NKnk4 w - - 0 1
Key: 3FF7F318447D441F
Checkers:
Tablebases WDL: -2 (1)
Tablebases DTZ: -12 (1)
Tablebases DTM: mate 0 (0)
(Should be DTZ -100)

[noobpwnftw]

The generator compile was correct, I checked.

Also, all 5v2 pawnless built, starting 5v2 pawnful with fixed WDL probing code. 4v3 pawnless also started, I think if anything happens it should only affect 16-bit tables.

[niklasf]

So far so good. This also explains why I was seeing correct WDL's in Python, where integers never overflow, as opposed to Cfish.

[noobpwnftw]

I may have found the issue(see PR), regenerating the table.

[syzygy]

I may have found the issue(see PR), regenerating the table.

Yes, that may be it.

[noobpwnftw]

I fear about this "len" being 1 there though. I think it is correct but not sure what it contributes to the "numpos + t > 65536" bounds check or other places.

[syzygy]

I may have found the issue(see PR), regenerating the table.

Yes, that may be it.

That earlier fix indeed skipped something important. I should have verified that change by generating a table with a lowered MAX_NARROW value.

The more recent change seems to work correctly as far as I can tell. So hopefully this covers everything now. (But there will probably be more surprises )

[syzygy]

I fear about this "len" being 1 there though. I think it is correct but not sure what it contributes to the "numpos + t > 65536" bounds check or other places.

A compressed block of data may cover at most 65536 positions. This is because the index structure of the table uses a 16-bit number to keep track of the number of positions in each block, where 0 means 1 position and 65535 means 65536 positions.

So here numpos counts the number of positions already stored in the block and t is the number of positions represented by the next symbol.

At the same time, bits counts the number of Huffman-compressed bits in the block and c->length[s] is the length of the Huffman code for the next symbol.

So the line

Code: Select all

    if (bits + c->length[s] > maxbits || numpos + t > 65536) {

tests whether the block is full and a new block has to be started.

The next lines (in write_ctb_data) try to squeeze a last symbol into the block if the limit of 65536 has not yet been reached. This is possible, for example, if a shorter huffman code exists representing a longer pattern with the same initial pattern. This can easily be the case, since each symbol consists either of the value of a single position or of two other symbols. If AA is almost always followed by BB, it could be that AA after all AABB strings have been removed occurs less frequently than AABB. The Huffman code for AA may then be longer than the Huffman code for AABB. If we are at the end of the block, we can then just encode AABB. The BB values will never be retrieved.

[noobpwnftw]

Oh, I see. Thank you for the explanation!

[Nordlandia]

Soon approaching 1 terabyte of generated data. Milestone!