Crafty 22.0 SMP on Windows

[maschmdt]

Hello,

when I tested the new Crafty 22.0 under the Chessbase Gui (with Wb2UCI) it didn't reach even the first position of the Marathon test. So I turned on the log mode of WB2Uci and discovered a command sequence wich crashes crafty. I was able to reproduce this with the crafty console like this:

White(1): mt 2
max threads set to 2
White(1): anaylze
Analyze Mode: type "exit" to terminate.
time surplus 0.00 time limit 57.00 (+27.00) (3:30)
depth time score variation (1)
starting thread 1
10 0.11 0.16 1. d4 Nf6 2. Nc3 d5 3. Nf3 Bf5 4. Bf4
...
exit
White(1): exit
analyze.White(1): analyze complete.
White(1): new

and Crafty quits silently

Next step was that I looked for changes in lock.h and thread.h since crafty 21.6.

In thread.c

Code: Select all

/*
 ************************************************************
 *                                                          *
 *   if we are waiting on a block and the busy count is now *
 *   zero, we simply return to finish up the bookkeeping at *
 *   that point.                                            *
 *                                                          *
 ************************************************************
 */
    Unlock(shared->lock_smp);
    if (shared->thread[tid] == waiting)
      return (0);
    if (shared->quit || shared->thread[tid] == (TREE *) - 1) {
      Lock(shared->lock_io);
      shared->smp_threads--;
      Unlock(shared->lock_io);
     exit(0);
    }

This was changed from return(0) in v21.6 to exit(0) in v22.0. I reverted that to return(0) compiled crafty and it seemed to work now but while running the marathon test again (210 tactical positions) crafty consumed more and more memory and crashed enventually around pos 100. So it seems to me that a return(0) at that point triggers a memory leak under Windows.

Sorry that I can't present a sound solution but it's a starting point.

Btw I used the MS compiler that comes with the Windows Server 2003 R2 Platform SDK. Os is Win XP64.

Regards,
Michael

[Guetti]

Hello,

when I tested the new Crafty 22.0 under the Chessbase Gui (with Wb2UCI) it didn't reach even the first position of the Marathon test. So I turned on the log mode of WB2Uci and discovered a command sequence wich crashes crafty. I was able to reproduce this with the crafty console like this:

White(1): mt 2
max threads set to 2
White(1): anaylze
Analyze Mode: type "exit" to terminate.
time surplus 0.00 time limit 57.00 (+27.00) (3:30)
depth time score variation (1)
starting thread 1
10 0.11 0.16 1. d4 Nf6 2. Nc3 d5 3. Nf3 Bf5 4. Bf4
...
exit
White(1): exit
analyze.White(1): analyze complete.
White(1): new

and Crafty quits silently

Next step was that I looked for changes in lock.h and thread.h since crafty 21.6.

In thread.c

Code: Select all

/*
 ************************************************************
 *                                                          *
 *   if we are waiting on a block and the busy count is now *
 *   zero, we simply return to finish up the bookkeeping at *
 *   that point.                                            *
 *                                                          *
 ************************************************************
 */
    Unlock(shared->lock_smp);
    if (shared->thread[tid] == waiting)
      return (0);
    if (shared->quit || shared->thread[tid] == (TREE *) - 1) {
      Lock(shared->lock_io);
      shared->smp_threads--;
      Unlock(shared->lock_io);
     exit(0);
    }

This was changed from return(0) in v21.6 to exit(0) in v22.0. I reverted that to return(0) compiled crafty and it seemed to work now but while running the marathon test again (210 tactical positions) crafty consumed more and more memory and crashed enventually around pos 100. So it seems to me that a return(0) at that point triggers a memory leak under Windows.

Sorry that I can't present a sound solution but it's a starting point.

Btw I used the MS compiler that comes with the Windows Server 2003 R2 Platform SDK. Os is Win XP64.

Regards,
Michael

exit(0) terminates a process. Crafty uses processes for SMP search, and if you remove this exit() (or change to return), then you probably continuously start to spawn new processes but never terminate one, that is why Crafty uses more and more memory, until either all your memory is gone, or MAX_PROCESSES of you OS is reached. Seems that your correction introduced a bug.

By the way, I could not reproduce the behaviour with crafty-22.0 under Linux. Here is what happens:

Code: Select all

Crafty v22.0 (1 cpus)

White(1): mt 2
max threads set to 2
White(1): book off
book file disabled.
White(1): analyze
Analyze Mode: type "exit" to terminate.
              time surplus   0.00  time limit 22.50 (+0.00) (3:30)
              depth   time  score   variation (1)
starting thread 1
               10     0.26   0.25   1. d4 Nf6 2. Nc3 Nc6 3. d5 Ne5 4. Nf3
                                    d6 5. e4 Bg4
               10->   0.28   0.25   1. d4 Nf6 2. Nc3 Nc6 3. d5 Ne5 4. Nf3
                                    d6 5. e4 Bg4
               11     0.43   0.14   1. d4 Nf6 2. Nc3 e6 3. e4 Bb4 4. Bg5
                                    h6 5. Bxf6 Bxc3+ 6. bxc3 Qxf6
               11     0.55   0.19   1. Nf3 Nf6 2. Nc3 d5 3. d3 Qd6 4. e4
                                    e5 5. Bg5 Nbd7 6. Nxd5 Nxd5 7. exd5
                                    Qxd5
               11->   0.67   0.19   1. Nf3 Nf6 2. Nc3 d5 3. d3 Qd6 4. e4
                                    e5 5. Bg5 Nbd7 6. Nxd5 Nxd5 7. exd5
                                    Qxd5 (s=3)
               12     0.86   0.18   1. Nf3 Nf6 2. Nc3 d5 3. e3 e6 4. d4
                                    Bb4 5. Bd3 Ne4 6. Bd2 Nxd2 7. Qxd2
                                    (s=2)
               12     1.30   0.25   1. e4 Nc6 2. Nf3 e6 3. Nc3 Nf6 4. d4
                                    <HT>
               12->   1.36   0.25   1. e4 Nc6 2. Nf3 e6 3. Nc3 Nf6 4. d4
                                    <HT> (s=2)
               13     2.02   0.28   1. e4 Nc6 2. Nf3 e6 3. Nc3 d5 4. exd5
                                    exd5 5. d4 Qe7+ 6. Be2 Nf6 7. O-O Bf5
               13->   2.32   0.28   1. e4 Nc6 2. Nf3 e6 3. Nc3 d5 4. exd5
                                    exd5 5. d4 Qe7+ 6. Be2 Nf6 7. O-O Bf5
               14     5.46   0.27   1. e4 e5 2. Nf3 Nf6 3. Nc3 Bb4 4. Nxe5
                                    Qe7 5. Ng4 Nxe4 6. Nd5 Qc5 7. Nxb4
                                    Qxb4
               14->   6.10   0.27   1. e4 e5 2. Nf3 Nf6 3. Nc3 Bb4 4. Nxe5
                                    Qe7 5. Ng4 Nxe4 6. Nd5 Qc5 7. Nxb4
                                    Qxb4
it             15     6.10   1/20?  1. e4      (2.1Mnps)             
White(1): exit
terminating SMP processes.analyze.White(1): analyze complete.
White(1): new
parallel threads terminated.
White(1): analyze
Analyze Mode: type "exit" to terminate.
              time surplus   0.00  time limit 22.50 (+0.00) (3:30)
              depth   time  score   variation (1)
starting thread 1
               10     0.29   0.15   1. e4 Nc6 2. Nc3 Nf6 3. Nf3 d5 4. e5
                                    Ne4 5. Nxe4 dxe4
               10->   0.33   0.15   1. e4 Nc6 2. Nc3 Nf6 3. Nf3 d5 4. e5
                                    Ne4 5. Nxe4 dxe4
               11     0.47   0.22   1. e4 Nc6 2. Nc3 Nf6 3. Nf3 e6 4. d4
                                    d5 5. Qd3 Bb4 6. e5
               11->   0.52   0.22   1. e4 Nc6 2. Nc3 Nf6 3. Nf3 e6 4. d4
                                    d5 5. Qd3 Bb4 6. e5
               12     0.93   0.31   1. e4 Nc6 2. Nf3 e6 3. Nc3 d5 4. d4
                                    dxe4 5. Nxe4 Nf6 6. Nxf6+ Qxf6 7. Bg5
                                    <HT>

For me it looks like the process termination method of crafty-22.0 currently is borked under Windows, which would be consistent with problems others have reported.
But surely Prof. Hyatt can tell you more.

[maschmdt]

[...]

exit(0) terminates a process. Crafty uses processes for SMP search, and if you remove this exit() (or change to return), then you probably continuously start to spawn new processes but never terminate one, that is why Crafty uses more and more memory, until either all your memory is gone, or MAX_PROCESSES of you OS is reached. Seems that your correction introduced a bug.

[...]


For me it looks like the process termination method of crafty-22.0 currently is borked under Windows, which would be consistent with problems others have reported.
But surely Prof. Hyatt can tell you more.

Yes exit(0) terminates a process. But the Windows build of crafty seems to use threads for the parallel search if the machine is not a NUMA type system. See utilty.c:

Code: Select all

pthread_t NumaStartThread(void *func, void *args)
{
  HANDLE hThread;
  ULONGLONG ullMask;

  WinNumaInit();
  if (fSystemIsNUMA) {
    ulNumaNode++;
    if (ulNumaNode > ulNumaNodes)
      ulNumaNode = 0;
    ullMask = ullProcessorMask[ulNumaNode];
    printf("Starting thread on node %d CPU mask %I64d\n", ulNumaNode, ullMask);
    SetThreadAffinityMask(GetCurrentThread(), (DWORD_PTR) ullMask);
    hThread = (HANDLE) _beginthreadex(0, 0, func, args, CREATE_SUSPENDED, 0);
    SetThreadAffinityMask(hThread, (DWORD_PTR) ullMask);
    ResumeThread(hThread);
    SetThreadAffinityMask(GetCurrentThread(), ullProcessorMask[0]);
  } else
    hThread = (HANDLE) _beginthreadex(0, 0, func, args, 0, 0);
  return hThread;
}

I tried "_endthreadex(0)" instead of exit(0) in thread.c but that doesn't the trick either since it does no destroy the thread object. Furthremore it seems that crafty does not take care of the thread handle:

iterate.c where parallel threads/processes are started:

Code: Select all

#if defined(_WIN32) || defined(_WIN64)
          NumaStartThread(ThreadInit, (void *) proc);
#else....

Maybe this should be a thead pool but this needs discussion.

Regards,
Michael

[Jim Ablett]

I had a try >

Code changes in 'thread.c'

Code: Select all

/*
 ************************************************************
 *                                                          *
 *   we can exit if our thread[i] is non-zero, or if we are *
 *   waiting on others to finish a block that *we* have to  *
 *   return through.  when the busy count on such a block   *
 *   hits zero, we return immediately which unwinds the     *
 *   search as it should be.                                *
 *                                                          *
 ************************************************************
 */
    
    
    while (!shared->thread[tid] && !shared->quit && (!waiting ||
            waiting->nprocs))
    if (shared->quit) {
      return (0);
    
}
  
  
    Lock(shared->lock_smp);
    if (!shared->thread[tid])
      shared->thread[tid] = waiting;

and here >

Code: Select all

/*
 ************************************************************
 *                                                          *
 *   if we are waiting on a block and the busy count is now *
 *   zero, we simply return to finish up the bookkeeping at *
 *   that point.                                            *
 *                                                          *
 ************************************************************
 */
  
    
    Unlock(shared->lock_smp);
    if (shared->thread[tid] == waiting)
      return (0);
    if (shared->quit || shared->thread[tid] == (TREE *) - 1) {
      Lock(shared->lock_io);
      Print(128, "thread %d exiting\n", tid);
      Unlock(shared->lock_io);
      return (0);
    }

    

I compiled a mingw-gcc build and tested it on my Core2Duo using crafty.rc settings -

Code: Select all

mt=2   &   smpnice=1.

Seems to be working properly, terminating it's thread and not using cpu when it's not its turn, but I'd like a second opinion of course.

Download test build here > http://www.zshare.net/download/8793429f6fe301/

Jim.

[bob]

[...]

exit(0) terminates a process. Crafty uses processes for SMP search, and if you remove this exit() (or change to return), then you probably continuously start to spawn new processes but never terminate one, that is why Crafty uses more and more memory, until either all your memory is gone, or MAX_PROCESSES of you OS is reached. Seems that your correction introduced a bug.

[...]


For me it looks like the process termination method of crafty-22.0 currently is borked under Windows, which would be consistent with problems others have reported.
But surely Prof. Hyatt can tell you more.

Yes exit(0) terminates a process. But the Windows build of crafty seems to use threads for the parallel search if the machine is not a NUMA type system. See utilty.c:

Code: Select all

pthread_t NumaStartThread(void *func, void *args)
{
  HANDLE hThread;
  ULONGLONG ullMask;

  WinNumaInit();
  if (fSystemIsNUMA) {
    ulNumaNode++;
    if (ulNumaNode > ulNumaNodes)
      ulNumaNode = 0;
    ullMask = ullProcessorMask[ulNumaNode];
    printf("Starting thread on node %d CPU mask %I64d\n", ulNumaNode, ullMask);
    SetThreadAffinityMask(GetCurrentThread(), (DWORD_PTR) ullMask);
    hThread = (HANDLE) _beginthreadex(0, 0, func, args, CREATE_SUSPENDED, 0);
    SetThreadAffinityMask(hThread, (DWORD_PTR) ullMask);
    ResumeThread(hThread);
    SetThreadAffinityMask(GetCurrentThread(), ullProcessorMask[0]);
  } else
    hThread = (HANDLE) _beginthreadex(0, 0, func, args, 0, 0);
  return hThread;
}

I tried "_endthreadex(0)" instead of exit(0) in thread.c but that doesn't the trick either since it does no destroy the thread object. Furthremore it seems that crafty does not take care of the thread handle:

iterate.c where parallel threads/processes are started:

Code: Select all

#if defined(_WIN32) || defined(_WIN64)
          NumaStartThread(ThreadInit, (void *) proc);
#else....

Maybe this should be a thead pool but this needs discussion.

Regards,
Michael

OK, that may well be a problem. I didn't write the windows code to create parallel processes (or threads). If it is really using threads, which seems wrong, then this new version is not going to work well at all. However, if it does use threads, then it would seem that some of the shared memory stuff would break. I will try to see what is going on and perhaps get a windows thread expert to help...

That might be why some have been reporting errors but I can't reproduce any of them...

[bob]

Jim:

would you try something for me, since you seem to have a handle on the windows side of things... determine if there is a safe way to terminate a thread, similar to what exit() does for a process. The threads really do need to terminate and not return, because there is a potential for a hang since the return might not take the thread back to the thread create routine where it will exit, it might take it back to an older split point that is no longer being searched, which can hang threads. That's why the exit() was done, it was causing problems to return under linux.

What I need is a "shoot the thread in the head" type of call that terminates it on the spot, then we can do a conditional test to do that for windoes and exit() for unix...

[Jim Ablett]

Hi Bob,

What I need is a "shoot the thread in the head" type of call that terminates it on the spot, then we can do a conditional test to do that for windoes and exit() for unix...

ExitThread() or TerminateThread() ?

http://msdn2.microsoft.com/en-us/librar ... S.85).aspx
http://msdn2.microsoft.com/en-us/librar ... S.85).aspx

Jim.

[Denis P. Mendoza]

I had a try >

Code changes in 'thread.c'

Code: Select all

/*
 ************************************************************
 *                                                          *
 *   we can exit if our thread[i] is non-zero, or if we are *
 *   waiting on others to finish a block that *we* have to  *
 *   return through.  when the busy count on such a block   *
 *   hits zero, we return immediately which unwinds the     *
 *   search as it should be.                                *
 *                                                          *
 ************************************************************
 */
    
    
    while (!shared->thread[tid] && !shared->quit && (!waiting ||
            waiting->nprocs))
    if (shared->quit) {
      return (0);
    
}
  
  
    Lock(shared->lock_smp);
    if (!shared->thread[tid])
      shared->thread[tid] = waiting;

and here >

Code: Select all

/*
 ************************************************************
 *                                                          *
 *   if we are waiting on a block and the busy count is now *
 *   zero, we simply return to finish up the bookkeeping at *
 *   that point.                                            *
 *                                                          *
 ************************************************************
 */
  
    
    Unlock(shared->lock_smp);
    if (shared->thread[tid] == waiting)
      return (0);
    if (shared->quit || shared->thread[tid] == (TREE *) - 1) {
      Lock(shared->lock_io);
      Print(128, "thread %d exiting\n", tid);
      Unlock(shared->lock_io);
      return (0);
    }

    

I compiled a mingw-gcc build and tested it on my Core2Duo using crafty.rc settings -

Code: Select all

mt=2   &   smpnice=1.

Seems to be working properly, terminating it's thread and not using cpu when it's not its turn, but I'd like a second opinion of course.

Download test build here > http://www.zshare.net/download/8793429f6fe301/

Jim.

Thanks Jim for the revision. I think it works fine as it doesn't hang anymore when you interrupt and resume games. SMP-Nice works fine Master Jim!

I made an Intel compile (32-bit) which reacts in the same way using your code revisions. I attached the fixed code for others to test.

http://www.zshare.net/download/8821895050ad97/

[Denis P. Mendoza]

This is an Intel 64-bit build for the fix:

http://www.zshare.net/download/8824395f0e2a0f/

[beachknight]

Hi Dennis,

Still buggy.

Typing bench or analyze and d4 on console mode
causes terminating Crafty.

Best,

PS: Jims compile runs fine, but with much lower performance.