Another Crafty-23.1 crash

Discussion of anything and everything relating to chess playing software and machines.

Moderator: Ras

zullil
Posts: 6442
Joined: Tue Jan 09, 2007 12:31 am
Location: PA USA
Full name: Louis Zulli

Another Crafty-23.1 crash

Post by zullil »

Again, during a game against Crafty-23.0 under xboard. The crash seems "identical" to the last one, at least according to the backtrace in the crash report:

Code: Select all

Process:         crafty-23.1 [1681]
Path:            /Users/Louis/Documents/Chess/Crafty/Crafty-23.1/crafty-23.1
Identifier:      crafty-23.1
Version:         ??? (???)
Code Type:       X86-64 (Native)
Parent Process:  xboard [1680]

Date/Time:       2009-12-11 11:44:10.977 -0500
OS Version:      Mac OS X 10.5.8 (9L30)
Report Version:  6
Anonymous UUID:  4CF52182-2BFE-41B2-8CD5-77275A03C082

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x000000000000be58
Crashed Thread:  7

Thread 0:
0   crafty-23.1                   	0x0000000100047573 ThreadWait + 83
1   crafty-23.1                   	0x0000000100047417 Thread + 1575
2   crafty-23.1                   	0x0000000100012a9e Iterate + 2686
3   crafty-23.1                   	0x0000000100001f13 main + 2611
4   crafty-23.1                   	0x00000001000014b4 start + 52

Thread 1:
0   crafty-23.1                   	0x0000000100047573 ThreadWait + 83
1   libSystem.B.dylib             	0x00007fff83f9fd4d thread_start + 13

Thread 2:
0   crafty-23.1                   	0x0000000100047573 ThreadWait + 83
1   libSystem.B.dylib             	0x00007fff83f9fd4d thread_start + 13

Thread 3:
0   crafty-23.1                   	0x0000000100047573 ThreadWait + 83
1   libSystem.B.dylib             	0x00007fff83f9fd4d thread_start + 13

Thread 4:
0   crafty-23.1                   	0x0000000100047581 ThreadWait + 97
1   libSystem.B.dylib             	0x00007fff83f9fd4d thread_start + 13

Thread 5:
0   crafty-23.1                   	0x0000000100047573 ThreadWait + 83
1   libSystem.B.dylib             	0x00007fff83f9fd4d thread_start + 13

Thread 6:
0   crafty-23.1                   	0x0000000100047573 ThreadWait + 83
1   libSystem.B.dylib             	0x00007fff83f9fd4d thread_start + 13

Thread 7 Crashed:
0   crafty-23.1                   	0x000000010003190b SearchParallel + 379
1   crafty-23.1                   	0x000000010004762e ThreadWait + 270
2   crafty-23.1                   	0x0000000100020bcd ThreadInit + 333
3   libSystem.B.dylib             	0x00007fff83f9fe8b _pthread_start + 316
4   libSystem.B.dylib             	0x00007fff83f9fd4d thread_start + 13

Thread 7 crashed with X86 Thread State (64-bit):
  rax: 0x000000000000001c  rbx: 0x0000000102420000  rcx: 0x0000000000000001  rdx: 0x000000000000be58
  rdi: 0x0000000000000000  rsi: 0x0000000000000000  rbp: 0x0000000000000e00  rsp: 0x000000010389dcf0
   r8: 0x0000000000000000   r9: 0x0000000000000000  r10: 0x0000000000000000  r11: 0x0000000000000001
  r12: 0x0000000102420000  r13: 0x0000000000000001  r14: 0x0000000102420000  r15: 0x0000000000000007
  rip: 0x000000010003190b  rfl: 0x0000000000010202  cr2: 0x000000000000be58

Binary Images:
       0x100000000 -        0x1000c7ff7 +crafty-23.1 ??? (???) <0a452c13de066031f69df9065efbd873> /Users/Louis/Documents/Chess/Crafty/Crafty-23.1/crafty-23.1
       0x1002c4000 -        0x1002c4ffd  libmx.A.dylib ??? (???) /usr/lib/libmx.A.dylib
    0x7fff5fc00000 -     0x7fff5fc2e643  dyld 97.1 (???) <1d1ba42c89e77cfe2558a3c66129fff6> /usr/lib/dyld
    0x7fff81f8b000 -     0x7fff81f8ffff  libmathCommon.A.dylib ??? (???) /usr/lib/system/libmathCommon.A.dylib
    0x7fff835fa000 -     0x7fff8366efe7  libstdc++.6.dylib ??? (???) <4c052234501252589cb46a124abc72d9> /usr/lib/libstdc++.6.dylib
    0x7fff83f3d000 -     0x7fff83f49ff1  libgcc_s.1.dylib ??? (???) <568895f7479fefc4796748a618c6ec9f> /usr/lib/libgcc_s.1.dylib
    0x7fff83f71000 -     0x7fff840fcffb  libSystem.B.dylib ??? (???) <714d2608b5acae3ad5364897c49868fa> /usr/lib/libSystem.B.dylib
    0x7fffffe00000 -     0x7fffffe01780  libSystem.B.dylib ??? (???) /usr/lib/libSystem.B.dylib

 
The function SearchParallel crashed. The crash was caused by a data access into unmapped memory at address 0x000000000000be58.

I haven't added that Print statement---should I still bother?---but here's the latest log file, with all but the top and bottom omitted:

Code: Select all

max threads set to 8.
EGTB access enabled
using tbpath=/Users/louis/Documents/Chess/Crafty/TB
5 piece tablebase files found
14043kb of RAM used for TB indices and decompression tables
EGTB cache memory =  256M bytes.
hash table memory = 1024M bytes.
pawn hash table memory =  512M bytes.
choose from book moves randomly (using weights.)
choose from 5 best moves.
pondering disabled.
Audio output disabled
 game/10 minutes primary time control


Crafty v23.1 (8 cpus)

White(1): xboard
White(1): protover 2
feature ping=1 setboard=1 san=1 time=1 draw=1
feature sigint=0 sigterm=0 reuse=1 analyze=1
feature myname="Crafty-23.1" name=1
feature playother=1 colors=0
feature variants="normal,nocastle"
feature done=1
White(1): accepted ping
White(1): accepted setboard
White(1): accepted san
White(1): accepted time
White(1): accepted draw
White(1): accepted sigint
White(1): accepted sigterm
White(1): accepted reuse
White(1): accepted analyze
White(1): accepted myname
White(1): accepted name
White(1): accepted playother
White(1): accepted colors
White(1): accepted variants
White(1): accepted done
White(1): new
parallel threads terminated.
White(1):  smpmt=8
max threads set to 8.
White(1):  learn=0
book learning disabled
White(1):  egtb
White(1):  cache=256M
EGTB cache memory =  256M bytes.
White(1):  hash=1024M
hash table memory = 1024M bytes.
White(1):  hashp=512M
pawn hash table memory =  512M bytes.
White(1):  book random 1
choose from book moves randomly (using weights.)
White(1):  book width 5
choose from 5 best moves.
White(1):  ponder=off
pondering disabled.
White(1):  xboard
White(1): level 0 3 0
 game/180 seconds primary time control
White(1): hard
pondering enabled.
White(1): easy
pondering disabled.
White(1): ping 1
pong 1
White(1): force
White(1): computer
playing a computer!
White(1): name Crafty-23.0
Crafty 23.1 vs Crafty-23.0
White(1): time 18000
time remaining:   3:00 (Crafty).
White(1): otim 18000
time remaining:   3:00 (opponent).
White(1): go
               book moves {e4, f4, d4}
               book   0.0s     38%    d4! Nf6
White(1): d4
              time used:   0.00
Black(1): time 17999
time remaining:   2:59 (Crafty).
Black(1): otim 17836
time remaining:   2:58 (opponent).
Black(1): Nf6
              time used:   1.63
               book moves {c4, Nf3}
               book   0.0s     79%    c4 e6



Black(41): a5
              time used:   4.12
              time limit 2.06 (+0.00) (12.36)
              depth   time  score   variation (1)
starting thread 1
starting thread 2
starting thread 3
starting thread 4
starting thread 5
starting thread 6
starting thread 7
               15     0.44   0.12   42. Qe1 Qb7+ 43. f3 Qa6 44. bxa5 Qxa5
                                    45. Qxa5 bxa5 46. c5 Ke7 47. Kf2 Kd7
                                    48. Ke3 Kc6 49. Kd4 e5+ 50. Kc4
               15->   0.83   0.12   42. Qe1 Qb7+ 43. f3 Qa6 44. bxa5 Qxa5
                                    45. Qxa5 bxa5 46. c5 Ke7 47. Kf2 Kd7
                                    48. Ke3 Kc6 49. Kd4 e5+ 50. Kc4
               16     2.14   0.09   42. Qe1 Qb7+ 43. Kg1 a4 44. Qe3 Qc6
                                    45. Qd3 e5 46. f3 Qe6 47. Kg2 f5 48.
                                    h4 h6 49. Kf2 Kf6 50. Qd8+ Kf7
               16->   3.76   0.09   42. Qe1 Qb7+ 43. Kg1 a4 44. Qe3 Qc6
                                    45. Qd3 e5 46. f3 Qe6 47. Kg2 f5 48.
                                    h4 h6 49. Kf2 Kf6 50. Qd8+ Kf7 (s=2)
              time=3.76  mat=0  n=66676617  fh=91%  nps=17.7M
              extensions=4.2M qchecks=4.6M reduced=6.1M pruned=9.7M
              predicted=23  evals=36.5M  50move=0  EGTBprobes=0  hits=0
              SMP->  splits=14408  aborts=1657  data=35/512  elap=3.76
terminating SMP processes.
White(42): Qe1
              time used:   3.76
Black(42): time 5622
time remaining:  56.22 (Crafty).
Black(42): otim 11102
time remaining:   1:51 (opponent).
Black(42): Qc7
              time used:   2.51
              time limit 1.93 (+0.00) (11.58)
              depth   time  score   variation (1)
starting thread 1
starting thread 2
starting thread 3
starting thread 4
starting thread 5
starting thread 6
starting thread 7

bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Another Crafty-23.1 crash

Post by bob »

zullil wrote:Again, during a game against Crafty-23.0 under xboard. The crash seems "identical" to the last one, at least according to the backtrace in the crash report:

Code: Select all

Process:         crafty-23.1 [1681]
Path:            /Users/Louis/Documents/Chess/Crafty/Crafty-23.1/crafty-23.1
Identifier:      crafty-23.1
Version:         ??? (???)
Code Type:       X86-64 (Native)
Parent Process:  xboard [1680]

Date/Time:       2009-12-11 11:44:10.977 -0500
OS Version:      Mac OS X 10.5.8 (9L30)
Report Version:  6
Anonymous UUID:  4CF52182-2BFE-41B2-8CD5-77275A03C082

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x000000000000be58
Crashed Thread:  7

Thread 0:
0   crafty-23.1                   	0x0000000100047573 ThreadWait + 83
1   crafty-23.1                   	0x0000000100047417 Thread + 1575
2   crafty-23.1                   	0x0000000100012a9e Iterate + 2686
3   crafty-23.1                   	0x0000000100001f13 main + 2611
4   crafty-23.1                   	0x00000001000014b4 start + 52

Thread 1:
0   crafty-23.1                   	0x0000000100047573 ThreadWait + 83
1   libSystem.B.dylib             	0x00007fff83f9fd4d thread_start + 13

Thread 2:
0   crafty-23.1                   	0x0000000100047573 ThreadWait + 83
1   libSystem.B.dylib             	0x00007fff83f9fd4d thread_start + 13

Thread 3:
0   crafty-23.1                   	0x0000000100047573 ThreadWait + 83
1   libSystem.B.dylib             	0x00007fff83f9fd4d thread_start + 13

Thread 4:
0   crafty-23.1                   	0x0000000100047581 ThreadWait + 97
1   libSystem.B.dylib             	0x00007fff83f9fd4d thread_start + 13

Thread 5:
0   crafty-23.1                   	0x0000000100047573 ThreadWait + 83
1   libSystem.B.dylib             	0x00007fff83f9fd4d thread_start + 13

Thread 6:
0   crafty-23.1                   	0x0000000100047573 ThreadWait + 83
1   libSystem.B.dylib             	0x00007fff83f9fd4d thread_start + 13

Thread 7 Crashed:
0   crafty-23.1                   	0x000000010003190b SearchParallel + 379
1   crafty-23.1                   	0x000000010004762e ThreadWait + 270
2   crafty-23.1                   	0x0000000100020bcd ThreadInit + 333
3   libSystem.B.dylib             	0x00007fff83f9fe8b _pthread_start + 316
4   libSystem.B.dylib             	0x00007fff83f9fd4d thread_start + 13

Thread 7 crashed with X86 Thread State (64-bit):
  rax: 0x000000000000001c  rbx: 0x0000000102420000  rcx: 0x0000000000000001  rdx: 0x000000000000be58
  rdi: 0x0000000000000000  rsi: 0x0000000000000000  rbp: 0x0000000000000e00  rsp: 0x000000010389dcf0
   r8: 0x0000000000000000   r9: 0x0000000000000000  r10: 0x0000000000000000  r11: 0x0000000000000001
  r12: 0x0000000102420000  r13: 0x0000000000000001  r14: 0x0000000102420000  r15: 0x0000000000000007
  rip: 0x000000010003190b  rfl: 0x0000000000010202  cr2: 0x000000000000be58

Binary Images:
       0x100000000 -        0x1000c7ff7 +crafty-23.1 ??? (???) <0a452c13de066031f69df9065efbd873> /Users/Louis/Documents/Chess/Crafty/Crafty-23.1/crafty-23.1
       0x1002c4000 -        0x1002c4ffd  libmx.A.dylib ??? (???) /usr/lib/libmx.A.dylib
    0x7fff5fc00000 -     0x7fff5fc2e643  dyld 97.1 (???) <1d1ba42c89e77cfe2558a3c66129fff6> /usr/lib/dyld
    0x7fff81f8b000 -     0x7fff81f8ffff  libmathCommon.A.dylib ??? (???) /usr/lib/system/libmathCommon.A.dylib
    0x7fff835fa000 -     0x7fff8366efe7  libstdc++.6.dylib ??? (???) <4c052234501252589cb46a124abc72d9> /usr/lib/libstdc++.6.dylib
    0x7fff83f3d000 -     0x7fff83f49ff1  libgcc_s.1.dylib ??? (???) <568895f7479fefc4796748a618c6ec9f> /usr/lib/libgcc_s.1.dylib
    0x7fff83f71000 -     0x7fff840fcffb  libSystem.B.dylib ??? (???) <714d2608b5acae3ad5364897c49868fa> /usr/lib/libSystem.B.dylib
    0x7fffffe00000 -     0x7fffffe01780  libSystem.B.dylib ??? (???) /usr/lib/libSystem.B.dylib

 
The function SearchParallel crashed. The crash was caused by a data access into unmapped memory at address 0x000000000000be58.

I haven't added that Print statement---should I still bother?---but here's the latest log file, with all but the top and bottom omitted:

Code: Select all

max threads set to 8.
EGTB access enabled
using tbpath=/Users/louis/Documents/Chess/Crafty/TB
5 piece tablebase files found
14043kb of RAM used for TB indices and decompression tables
EGTB cache memory =  256M bytes.
hash table memory = 1024M bytes.
pawn hash table memory =  512M bytes.
choose from book moves randomly (using weights.)
choose from 5 best moves.
pondering disabled.
Audio output disabled
 game/10 minutes primary time control


Crafty v23.1 (8 cpus)

White(1): xboard
White(1): protover 2
feature ping=1 setboard=1 san=1 time=1 draw=1
feature sigint=0 sigterm=0 reuse=1 analyze=1
feature myname="Crafty-23.1" name=1
feature playother=1 colors=0
feature variants="normal,nocastle"
feature done=1
White(1): accepted ping
White(1): accepted setboard
White(1): accepted san
White(1): accepted time
White(1): accepted draw
White(1): accepted sigint
White(1): accepted sigterm
White(1): accepted reuse
White(1): accepted analyze
White(1): accepted myname
White(1): accepted name
White(1): accepted playother
White(1): accepted colors
White(1): accepted variants
White(1): accepted done
White(1): new
parallel threads terminated.
White(1):  smpmt=8
max threads set to 8.
White(1):  learn=0
book learning disabled
White(1):  egtb
White(1):  cache=256M
EGTB cache memory =  256M bytes.
White(1):  hash=1024M
hash table memory = 1024M bytes.
White(1):  hashp=512M
pawn hash table memory =  512M bytes.
White(1):  book random 1
choose from book moves randomly (using weights.)
White(1):  book width 5
choose from 5 best moves.
White(1):  ponder=off
pondering disabled.
White(1):  xboard
White(1): level 0 3 0
 game/180 seconds primary time control
White(1): hard
pondering enabled.
White(1): easy
pondering disabled.
White(1): ping 1
pong 1
White(1): force
White(1): computer
playing a computer!
White(1): name Crafty-23.0
Crafty 23.1 vs Crafty-23.0
White(1): time 18000
time remaining:   3:00 (Crafty).
White(1): otim 18000
time remaining:   3:00 (opponent).
White(1): go
               book moves {e4, f4, d4}
               book   0.0s     38%    d4! Nf6
White(1): d4
              time used:   0.00
Black(1): time 17999
time remaining:   2:59 (Crafty).
Black(1): otim 17836
time remaining:   2:58 (opponent).
Black(1): Nf6
              time used:   1.63
               book moves {c4, Nf3}
               book   0.0s     79%    c4 e6



Black(41): a5
              time used:   4.12
              time limit 2.06 (+0.00) (12.36)
              depth   time  score   variation (1)
starting thread 1
starting thread 2
starting thread 3
starting thread 4
starting thread 5
starting thread 6
starting thread 7
               15     0.44   0.12   42. Qe1 Qb7+ 43. f3 Qa6 44. bxa5 Qxa5
                                    45. Qxa5 bxa5 46. c5 Ke7 47. Kf2 Kd7
                                    48. Ke3 Kc6 49. Kd4 e5+ 50. Kc4
               15->   0.83   0.12   42. Qe1 Qb7+ 43. f3 Qa6 44. bxa5 Qxa5
                                    45. Qxa5 bxa5 46. c5 Ke7 47. Kf2 Kd7
                                    48. Ke3 Kc6 49. Kd4 e5+ 50. Kc4
               16     2.14   0.09   42. Qe1 Qb7+ 43. Kg1 a4 44. Qe3 Qc6
                                    45. Qd3 e5 46. f3 Qe6 47. Kg2 f5 48.
                                    h4 h6 49. Kf2 Kf6 50. Qd8+ Kf7
               16->   3.76   0.09   42. Qe1 Qb7+ 43. Kg1 a4 44. Qe3 Qc6
                                    45. Qd3 e5 46. f3 Qe6 47. Kg2 f5 48.
                                    h4 h6 49. Kf2 Kf6 50. Qd8+ Kf7 (s=2)
              time=3.76  mat=0  n=66676617  fh=91%  nps=17.7M
              extensions=4.2M qchecks=4.6M reduced=6.1M pruned=9.7M
              predicted=23  evals=36.5M  50move=0  EGTBprobes=0  hits=0
              SMP->  splits=14408  aborts=1657  data=35/512  elap=3.76
terminating SMP processes.
White(42): Qe1
              time used:   3.76
Black(42): time 5622
time remaining:  56.22 (Crafty).
Black(42): otim 11102
time remaining:   1:51 (opponent).
Black(42): Qc7
              time used:   2.51
              time limit 1.93 (+0.00) (11.58)
              depth   time  score   variation (1)
starting thread 1
starting thread 2
starting thread 3
starting thread 4
starting thread 5
starting thread 6
starting thread 7

I'll see if I can fire up some 8-core vs 8-core tests to see if I can reproduce this at all. Just running one instance of Crafty is working perfectly. I play with 8 cores on ICC all the time and have not seen any crashes whatsoever there this year. Even some of my cluster testing has used 8 cores although I don't play Crafty vs Crafty nor do I use opening books on the cluster. I have a couple of million games using 8 cores and again, not a single crash. Not sure what to make of this. Could be a hardware issue (I have seen this on overclocked hardware) or a compiler issue, or some program bug that only shows up rarely, although it seems pretty frequent from your posts...
zullil
Posts: 6442
Joined: Tue Jan 09, 2007 12:31 am
Location: PA USA
Full name: Louis Zulli

Re: Another Crafty-23.1 crash

Post by zullil »

bob wrote: I'll see if I can fire up some 8-core vs 8-core tests to see if I can reproduce this at all. Just running one instance of Crafty is working perfectly. I play with 8 cores on ICC all the time and have not seen any crashes whatsoever there this year. Even some of my cluster testing has used 8 cores although I don't play Crafty vs Crafty nor do I use opening books on the cluster. I have a couple of million games using 8 cores and again, not a single crash. Not sure what to make of this. Could be a hardware issue (I have seen this on overclocked hardware) or a compiler issue, or some program bug that only shows up rarely, although it seems pretty frequent from your posts...
My hardware is all "standard." No over-clocking or other modifications. I compiled using icc. Here's the relevant portion of the makefile I used:

Code: Select all

darwin:
        $(MAKE) target=FreeBSD \
                CC=icc CXX=icpc \
                CFLAGS='-O3 -ipo -mdynamic-no-pic -static -no-prec-div -xHost -prof-use' \
                CXFLAGS='-O3 -ipo -mdynamic-no-pic -static -no-prec-div -xHost' \
                LDFLAGS=$(LDFLAGS) \
                LIBS='-lpthread -lstdc++' \
                opt='-DCPUS=8 -DINLINE64 -DLOGDIR="./Logs" \
                     -DBOOKDIR="/Users/louis/Documents/Chess/Crafty/Books/23" \
                     -DTBDIR="/Users/louis/Documents/Chess/Crafty/TB"' \
                crafty-make
As you can see, I used pgo. Maybe the crash is compiler-related. Thanks for investigating this as much as you have already.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Another Crafty-23.1 crash

Post by bob »

zullil wrote:
bob wrote: I'll see if I can fire up some 8-core vs 8-core tests to see if I can reproduce this at all. Just running one instance of Crafty is working perfectly. I play with 8 cores on ICC all the time and have not seen any crashes whatsoever there this year. Even some of my cluster testing has used 8 cores although I don't play Crafty vs Crafty nor do I use opening books on the cluster. I have a couple of million games using 8 cores and again, not a single crash. Not sure what to make of this. Could be a hardware issue (I have seen this on overclocked hardware) or a compiler issue, or some program bug that only shows up rarely, although it seems pretty frequent from your posts...
My hardware is all "standard." No over-clocking or other modifications. I compiled using icc. Here's the relevant portion of the makefile I used:

Code: Select all

darwin:
        $(MAKE) target=FreeBSD \
                CC=icc CXX=icpc \
                CFLAGS='-O3 -ipo -mdynamic-no-pic -static -no-prec-div -xHost -prof-use' \
                CXFLAGS='-O3 -ipo -mdynamic-no-pic -static -no-prec-div -xHost' \
                LDFLAGS=$(LDFLAGS) \
                LIBS='-lpthread -lstdc++' \
                opt='-DCPUS=8 -DINLINE64 -DLOGDIR="./Logs" \
                     -DBOOKDIR="/Users/louis/Documents/Chess/Crafty/Books/23" \
                     -DTBDIR="/Users/louis/Documents/Chess/Crafty/TB"' \
                crafty-make
As you can see, I used pgo. Maybe the crash is compiler-related. Thanks for investigating this as much as you have already.
I'm using ICC also and always use PGO with no problems. This still needs investigation. I have had quirks show up in older pthread libraries, whether this is related to that or not is unknown at present.