c++ class or a c wrapper for the syzygy egtb

[flok]

Some failing FEN-strings:

segfault:
8/R7/P4k2/8/8/8/8/2KP4 w - - 10 1

Well this is invalid position with pawn on rank 1
Edit: in fact, all of those positions you posted are invalid/illegal

Oh indeed. Darn.
It did now 250k tests and no crashes to be seen.
25k iterations under valgrind and no errors.

The "sanitizer" of gcc finds the following (in 7k iterations):

tbcore.c:1184:7: runtime error: load of misaligned address 0x7f5a6b4e303e for type 'int', which requires 4 byte alignment
0x7f5a6b4e303e: note: pointer points here
00 00 ff 0f 23 8e a0 00 d1 f9 c8 8f d1 23 60 ff 7f ef c6 8e 70 fe c4 fb b4 af 5f 06 59 83 fc b7
^
tbcore.c:1184:7: runtime error: load of misaligned address 0x7f5a6b4e5aa7 for type 'int', which requires 4 byte alignment
0x7f5a6b4e5aa7: note: pointer points here
f8 da bf fd ea 9f fe ba 3e f7 c6 5f fe a5 2f f6 38 bf f1 30 4f fe 0d df f0 48 ee f6 4d 5f fb df
^
tbcore.c:1184:7: runtime error: load of misaligned address 0x7f5a6b4e5ff3 for type 'int', which requires 4 byte alignment
0x7f5a6b4e5ff3: note: pointer points here
fe bb 1f f2 e0 4f f8 e0 2f f6 e0 9f f8 e7 2f f6 c3 1f f2 e7 3f f4 e0 1f f2 ae ff f9 e2 2f fe dc
^
tbcore.c:1184:7: runtime error: load of misaligned address 0x7f5a6b4e5fde for type 'int', which requires 4 byte alignment
0x7f5a6b4e5fde: note: pointer points here
fe d9 9f fd 02 f0 ff aa df fe f0 0f ff e2 ef fe be bf fe bb df fe bb 1f f2 e0 4f f8 e0 2f f6 e0
^


So I'm starting to think that the problems I see in my own program with syzygy are really bugs in my own program and not in the etb-code.
Odd because the perft-values are correct.

[mar]

tbcore.c:1184:7: runtime error: load of misaligned address 0x7f5a6b4e303e for type 'int', which requires 4 byte alignment
0x7f5a6b4e303e: note: pointer points here
00 00 ff 0f 23 8e a0 00 d1 f9 c8 8f d1 23 60 ff 7f ef c6 8e 70 fe c4 fb b4 af 5f 06 59 83 fc b7
^

Well, unaligned access is ok on x86/x64, not so on ARM where it's a show stopper.

[basil00]

I wrote a bit of code to trigger the problem.
My wrapper creates random fen-strings (well not entirely random; they're all (supposed to be-) valid but their position is random) and after only a couple of iterations it'll crash because of an assertion or a segfault.

Thanks for the investigation.

Edit: in fact, all of those positions you posted are invalid/illegal

Good catch.

The Fathom WDL probe code does not do any sanity checking since it is designed to be called during search, and thus should be as fast as possible. But maybe this can be made configurable somehow. The DTZ probe does do sanity checking, which is OK since it should only be called from the root.

So maybe Fathom is stable for 6-man after all? As mentioned, it was stress tested for 5-man but I do not currently have 6-man to test.

[basil00]

Well, unaligned access is ok on x86/x64, not so on ARM where it's a show stopper.

Also those warnings concern tbcore, which is basically unchanged from the original version from here: https://github.com/syzygy1/tb

[mvk]

tbcore.c:1184:7: runtime error: load of misaligned address 0x7f5a6b4e303e for type 'int', which requires 4 byte alignment
0x7f5a6b4e303e: note: pointer points here
00 00 ff 0f 23 8e a0 00 d1 f9 c8 8f d1 23 60 ff 7f ef c6 8e 70 fe c4 fb b4 af 5f 06 59 83 fc b7
^

Well, unaligned access is ok on x86/x64, not so on ARM where it's a show stopper.

I have this kind of things for arm:

Code: Select all

static inline
uint32 read_uint32(const ubyte *p)
{
  return &#40;uint32&#41;p&#91;0&#93; | (&#40;uint32&#41;p&#91;1&#93; << 8&#41; | (&#40;uint32&#41;p&#91;2&#93; << 16&#41; | (&#40;uint32&#41;p&#91;3&#93; << 24&#41;;
&#125;

…

  uint32 block = read_uint32&#40;&#40;unsigned char *&#41;d->indextable + 6 * mainidx&#41;;

tbcore.c also assumes char is unsigned. But it is unsigned on arm.

Code: Select all

 static const char offdiag[] = {
  0,-1,-1,-1,-1,-1,-1,-1,

And there is a (obscure) bug somewhere which clang finds but gcc doesn't. It was fixed in stockfish I think. I forgot where.

[flok]

Well, unaligned access is ok on x86/x64, not so on ARM where it's a show stopper.

Also those warnings concern tbcore, which is basically unchanged from the original version from here: https://github.com/syzygy1/tb

I found the problem with my fen creating (en-passant field incorrect). Now works fine for that.

I do get occasional random crashes. So valgrind to the rescue.
Valgrind says that there are data races.
I saw that the wdl code does locking but am I right that tb_pop_count() and tb_probe_root() don't do this at all?

Code: Select all

==6077== Possible data race during read of size 8 at 0x661FC0 by thread #1 
==6077== Locks held: none 
==6077==    at 0x4340B5: probe_dtz_table (tbprobe.c:773) 
==6077==    by 0x4340B5: probe_dtz_no_ep (tbprobe.c:1478) 
==6077==    by 0x4340B5: probe_dtz (tbprobe.c:1573) 
==6077==    by 0x434F0D: probe_root (tbprobe.c:1660) 
==6077==    by 0x436D65: tb_probe_root_impl (tbprobe.c:1861) 
==6077==    by 0x42ACFD: tb_probe_root (tbprobe.h:278) 
==6077==    by 0x42ACFD: query_syzygy_etb (fathom.c:661) 
==6077==    by 0x42160B: checkSyzygy(Scene*, PlayerColor, int*) (Brain.cpp:572)  
...

tbprobe.c:773

Code: Select all

    uint64_t key = calc_key(pos, false);
          
      if (DTZ_table[0].key1 != key && DTZ_table[0].key2 != key)
      {   
          for &#40;i = 1; i < DTZ_ENTRIES; i++)

(it's the if statement)

Code: Select all

==6077== This conflicts with a previous write of size 8 by thread #275 
==6077== Locks held: none 
==6077==    at 0x432763: load_dtz_table (tbcore.c:1559) 
==6077==    by 0x4341A8: probe_dtz_table (tbprobe.c:810) 
==6077==    by 0x4341A8: probe_dtz_no_ep (tbprobe.c:1478)
==6077==    by 0x4341A8: probe_dtz (tbprobe.c:1573)
==6077==    by 0x4351C2: probe_root (tbprobe.c:1686)
==6077==    by 0x436D65: tb_probe_root_impl (tbprobe.c:1861)
==6077==    by 0x42ACFD: tb_probe_root (tbprobe.h:278)
==6077==    by 0x42ACFD: query_syzygy_etb (fathom.c:661)
==6077==    by 0x42160B: checkSyzygy(Scene*, PlayerColor, int*) (Brain.cpp:572)

tbcore.c:1559

The key1-assignment.

Code: Select all

    struct TBHashEntry *ptr2;
  
    DTZ_table[0].key1 = key1;
    DTZ_table[0].key2 = key2;
    DTZ_table[0].entry = NULL;

[syzygy]

tbcore.c:1184:7: runtime error: load of misaligned address 0x7f5a6b4e303e for type 'int', which requires 4 byte alignment
0x7f5a6b4e303e: note: pointer points here
00 00 ff 0f 23 8e a0 00 d1 f9 c8 8f d1 23 60 ff 7f ef c6 8e 70 fe c4 fb b4 af 5f 06 59 83 fc b7
^

Well, unaligned access is ok on x86/x64, not so on ARM where it's a show stopper.

Correct.

In Stockfish the unaligned access has been worked around, at the same time eliminating a dependency on endianness:

Code: Select all

  ubyte *sympat = d->sympat;
  while (symlen[sym] != 0) {
    int w = *(int *)(sympat + 3 * sym);
    int s1 = w & 0x0fff;
    if &#40;litidx < &#40;int&#41;symlen&#91;s1&#93; + 1&#41;
      sym = s1;
    else &#123;
      litidx -= &#40;int&#41;symlen&#91;s1&#93; + 1;
      sym = &#40;w >> 12&#41; & 0x0fff;
    &#125;
  &#125;

became

Code: Select all

  ubyte *sympat = d->sympat;
  while (symlen[sym] != 0) {
    ubyte* w = sympat + (3 * sym);
    int s1 = (&#40;w&#91;1&#93; & 0xf&#41; << 8&#41; | w&#91;0&#93;;
    if &#40;litidx < &#40;int&#41;symlen&#91;s1&#93; + 1&#41;
      sym = s1;
    else &#123;
      litidx -= &#40;int&#41;symlen&#91;s1&#93; + 1;
      sym = &#40;w&#91;2&#93; << 4&#41; | &#40;w&#91;1&#93; >> 4&#41;;
    &#125;
  &#125;

A few more endianness dependencies were removed in setup_pairs() and decompress_pairs(). I think that is all.

[syzygy]

I saw that the wdl code does locking but am I right that tb_pop_count() and tb_probe_root() don't do this at all?

I don't know what tb_pop_count() does, but probe_root() should not be invoked in parallel. There should not be any need to either, as this function should only be invoked at the root position and not from within the search tree.

[basil00]

I don't know what tb_pop_count() does

Some basic functionality (such as population count) is exposed in order to make it easy to create stand-alone apps that use the syzygy tablebases. Currently there is only one such as (see fathom.c). The "tb_" prefix is just for consistency, i.e. all exported functions have the same prefix.

[stuwph]

I sometimes get a segfault in the syzygy code:

There is possibly some issue with 6-man (I could not test 6-man). Try recompiling with -fsanitize=address and see what it says if anything.

I could not guess the problem from the stack trace.

to get 6-men working at start of tbcore.c #define HSHMAX 5 needs to be changed to 6

which also relates to GULL syzygy ...