Crafty question: Multi GB hash

[Cardoso]

Hi,
I add another 6GB (3x2GB) set to my machine (corei7). I've now got 12GB RAM in windows 7 x64 professional.
In a crafty mod, and using VisualStudio 2010, when trying to allocate 2GB RAM my crafty compile gave the following error in HashProbe():
Unhandled exception at xxx adress,
0xC0000005: Access violation reading location 0x000000007dc23540.

Code: Select all

 for &#40;entry = 0; entry < 4; entry++, htable++) &#123;
    word1 = htable->word1;
    word2 = htable->word2 ^ word1; <-------- error here
    if &#40;word2 == temp_hashkey&#41;
      break;
 

I notice AllignedMalloc uses malloc.
What is the fix for this?
Maybe using VirtualMalloc ? If so what is the best way to implement Virtualalloc while keeping the hashtable alligned?

your comments will be much appreciated,
best regards,
Alvaro

[bob]

Hi,
I add another 6GB (3x2GB) set to my machine (corei7). I've now got 12GB RAM in windows 7 x64 professional.
In a crafty mod, and using VisualStudio 2010, when trying to allocate 2GB RAM my crafty compile gave the following error in HashProbe():
Unhandled exception at xxx adress,
0xC0000005: Access violation reading location 0x000000007dc23540.

Code: Select all

 for &#40;entry = 0; entry < 4; entry++, htable++) &#123;
    word1 = htable->word1;
    word2 = htable->word2 ^ word1; <-------- error here
    if &#40;word2 == temp_hashkey&#41;
      break;
 

I notice AllignedMalloc uses malloc.
What is the fix for this?
Maybe using VirtualMalloc ? If so what is the best way to implement Virtualalloc while keeping the hashtable alligned?

your comments will be much appreciated,
best regards,
Alvaro

Are you using a 64 bit compiler and a 64 bit operating system? Crafty has no "internal checks" for sanity of numbers like hash size. In looking at your note above, that makes no sense. How can "htable->word1" be perfectly OK, but "htable->word2" fail? In any case, here is Crafty running on a 12 gb machine with 8gb of hash:

White(1): hash=8192M
Warning-- xboard 'memory' option disabled
hash table memory = 8192M bytes (512M entries).
White(1): book off
book file disabled.
White(1): e4
time used: 27.04
time surplus 0.00 time limit 30.00 (+0.00) (3:00)
depth time score variation (1)
starting thread 1 2 3 4 5 6 7 8 9 10 11 <done>
16 0.48 0.20 1. ... Nc6 2. Nf3 e5 3. Nc3 Nf6 4.
Bc4 Bc5 5. O-O O-O 6. d3 d6 7. Bg5
h6 8. Be3 Be6 9. Qe2 Bxc4 10. Bxc5
dxc5 11. dxc4
16-> 0.59 0.20 1. ... Nc6 2. Nf3 e5 3. Nc3 Nf6 4.
Bc4 Bc5 5. O-O O-O 6. d3 d6 7. Bg5
h6 8. Be3 Be6 9. Qe2 Bxc4 10. Bxc5
dxc5 11. dxc4


So there is no reason I am aware of, EXCEPT for a version compiled with a 32 bit compiler, which will definitely break on hash sizes of 2gb and beyond because that turns into a negative number that will cause all sorts of problems.

[Cardoso]

Hi,
I add another 6GB (3x2GB) set to my machine (corei7). I've now got 12GB RAM in windows 7 x64 professional.
In a crafty mod, and using VisualStudio 2010, when trying to allocate 2GB RAM my crafty compile gave the following error in HashProbe():
Unhandled exception at xxx adress,
0xC0000005: Access violation reading location 0x000000007dc23540.

Code: Select all

 for &#40;entry = 0; entry < 4; entry++, htable++) &#123;
    word1 = htable->word1;
    word2 = htable->word2 ^ word1; <-------- error here
    if &#40;word2 == temp_hashkey&#41;
      break;
 

I notice AllignedMalloc uses malloc.
What is the fix for this?
Maybe using VirtualMalloc ? If so what is the best way to implement Virtualalloc while keeping the hashtable alligned?

your comments will be much appreciated,
best regards,
Alvaro

Are you using a 64 bit compiler and a 64 bit operating system? Crafty has no "internal checks" for sanity of numbers like hash size. In looking at your note above, that makes no sense. How can "htable->word1" be perfectly OK, but "htable->word2" fail? In any case, here is Crafty running on a 12 gb machine with 8gb of hash:

White(1): hash=8192M
Warning-- xboard 'memory' option disabled
hash table memory = 8192M bytes (512M entries).
White(1): book off
book file disabled.
White(1): e4
time used: 27.04
time surplus 0.00 time limit 30.00 (+0.00) (3:00)
depth time score variation (1)
starting thread 1 2 3 4 5 6 7 8 9 10 11 <done>
16 0.48 0.20 1. ... Nc6 2. Nf3 e5 3. Nc3 Nf6 4.
Bc4 Bc5 5. O-O O-O 6. d3 d6 7. Bg5
h6 8. Be3 Be6 9. Qe2 Bxc4 10. Bxc5
dxc5 11. dxc4
16-> 0.59 0.20 1. ... Nc6 2. Nf3 e5 3. Nc3 Nf6 4.
Bc4 Bc5 5. O-O O-O 6. d3 d6 7. Bg5
h6 8. Be3 Be6 9. Qe2 Bxc4 10. Bxc5
dxc5 11. dxc4


So there is no reason I am aware of, EXCEPT for a version compiled with a 32 bit compiler, which will definitely break on hash sizes of 2gb and beyond because that turns into a negative number that will cause all sorts of problems.

Thanks Bob for your feedback.
I really must have made a boboo somewhere.
Yes the compiler is VS2010 and a x64 compile and Windows 7 x64.
However I'm now sure I must made something wrong.
I'll have to investigate.

thanks again,
Alvaro

[bob]

Hi,
I add another 6GB (3x2GB) set to my machine (corei7). I've now got 12GB RAM in windows 7 x64 professional.
In a crafty mod, and using VisualStudio 2010, when trying to allocate 2GB RAM my crafty compile gave the following error in HashProbe():
Unhandled exception at xxx adress,
0xC0000005: Access violation reading location 0x000000007dc23540.

Code: Select all

 for &#40;entry = 0; entry < 4; entry++, htable++) &#123;
    word1 = htable->word1;
    word2 = htable->word2 ^ word1; <-------- error here
    if &#40;word2 == temp_hashkey&#41;
      break;
 

I notice AllignedMalloc uses malloc.
What is the fix for this?
Maybe using VirtualMalloc ? If so what is the best way to implement Virtualalloc while keeping the hashtable alligned?

your comments will be much appreciated,
best regards,
Alvaro

Are you using a 64 bit compiler and a 64 bit operating system? Crafty has no "internal checks" for sanity of numbers like hash size. In looking at your note above, that makes no sense. How can "htable->word1" be perfectly OK, but "htable->word2" fail? In any case, here is Crafty running on a 12 gb machine with 8gb of hash:

White(1): hash=8192M
Warning-- xboard 'memory' option disabled
hash table memory = 8192M bytes (512M entries).
White(1): book off
book file disabled.
White(1): e4
time used: 27.04
time surplus 0.00 time limit 30.00 (+0.00) (3:00)
depth time score variation (1)
starting thread 1 2 3 4 5 6 7 8 9 10 11 <done>
16 0.48 0.20 1. ... Nc6 2. Nf3 e5 3. Nc3 Nf6 4.
Bc4 Bc5 5. O-O O-O 6. d3 d6 7. Bg5
h6 8. Be3 Be6 9. Qe2 Bxc4 10. Bxc5
dxc5 11. dxc4
16-> 0.59 0.20 1. ... Nc6 2. Nf3 e5 3. Nc3 Nf6 4.
Bc4 Bc5 5. O-O O-O 6. d3 d6 7. Bg5
h6 8. Be3 Be6 9. Qe2 Bxc4 10. Bxc5
dxc5 11. dxc4


So there is no reason I am aware of, EXCEPT for a version compiled with a 32 bit compiler, which will definitely break on hash sizes of 2gb and beyond because that turns into a negative number that will cause all sorts of problems.

Thanks Bob for your feedback.
I really must have made a boboo somewhere.
Yes the compiler is VS2010 and a x64 compile and Windows 7 x64.
However I'm now sure I must made something wrong.
I'll have to investigate.

thanks again,
Alvaro

1. which version of Crafty? A few in the past have had large hash issues in windows.

2. If this is a 23.4 (most current version) and it is failing, let me know. There is a different malloc() procedure for windows vs unix and there might well be some sort of casting issue that screws up. I force the hash pointer to start on a 64-byte memory address boundary for cache efficiency. That can break if it is done incorrectly...

Let me know which version and maybe it is something that needs fixing.

[Graham Banks]

From Ray:

64-bit windows complies crash for me too with 2GB of hash. You could post that for me
1GB is OK
both Peter Skinner and JA compiles have the same problem

[bob]

From Ray:

64-bit windows complies crash for me too with 2GB of hash. You could post that for me
1GB is OK
both Peter Skinner and JA compiles have the same problem

I will give it a peek, later today...

[Graham Banks]

From Ray:

64-bit windows complies crash for me too with 2GB of hash. You could post that for me
1GB is OK
both Peter Skinner and JA compiles have the same problem

I will give it a peek, later today...

Thanks Bob. Happy New Year to you.

[Cardoso]

Hi,
I add another 6GB (3x2GB) set to my machine (corei7). I've now got 12GB RAM in windows 7 x64 professional.
In a crafty mod, and using VisualStudio 2010, when trying to allocate 2GB RAM my crafty compile gave the following error in HashProbe():
Unhandled exception at xxx adress,
0xC0000005: Access violation reading location 0x000000007dc23540.

Code: Select all

 for &#40;entry = 0; entry < 4; entry++, htable++) &#123;
    word1 = htable->word1;
    word2 = htable->word2 ^ word1; <-------- error here
    if &#40;word2 == temp_hashkey&#41;
      break;
 

I notice AllignedMalloc uses malloc.
What is the fix for this?
Maybe using VirtualMalloc ? If so what is the best way to implement Virtualalloc while keeping the hashtable alligned?

your comments will be much appreciated,
best regards,
Alvaro

Are you using a 64 bit compiler and a 64 bit operating system? Crafty has no "internal checks" for sanity of numbers like hash size. In looking at your note above, that makes no sense. How can "htable->word1" be perfectly OK, but "htable->word2" fail? In any case, here is Crafty running on a 12 gb machine with 8gb of hash:

White(1): hash=8192M
Warning-- xboard 'memory' option disabled
hash table memory = 8192M bytes (512M entries).
White(1): book off
book file disabled.
White(1): e4
time used: 27.04
time surplus 0.00 time limit 30.00 (+0.00) (3:00)
depth time score variation (1)
starting thread 1 2 3 4 5 6 7 8 9 10 11 <done>
16 0.48 0.20 1. ... Nc6 2. Nf3 e5 3. Nc3 Nf6 4.
Bc4 Bc5 5. O-O O-O 6. d3 d6 7. Bg5
h6 8. Be3 Be6 9. Qe2 Bxc4 10. Bxc5
dxc5 11. dxc4
16-> 0.59 0.20 1. ... Nc6 2. Nf3 e5 3. Nc3 Nf6 4.
Bc4 Bc5 5. O-O O-O 6. d3 d6 7. Bg5
h6 8. Be3 Be6 9. Qe2 Bxc4 10. Bxc5
dxc5 11. dxc4


So there is no reason I am aware of, EXCEPT for a version compiled with a 32 bit compiler, which will definitely break on hash sizes of 2gb and beyond because that turns into a negative number that will cause all sorts of problems.

Thanks Bob for your feedback.
I really must have made a boboo somewhere.
Yes the compiler is VS2010 and a x64 compile and Windows 7 x64.
However I'm now sure I must made something wrong.
I'll have to investigate.

thanks again,
Alvaro

1. which version of Crafty? A few in the past have had large hash issues in windows.

2. If this is a 23.4 (most current version) and it is failing, let me know. There is a different malloc() procedure for windows vs unix and there might well be some sort of casting issue that screws up. I force the hash pointer to start on a 64-byte memory address boundary for cache efficiency. That can break if it is done incorrectly...

Let me know which version and maybe it is something that needs fixing.

it's 23.4
but take notice I made some mods to be VS2010 friendly.

thanks,
Alvaro

[bob]

Hi,
I add another 6GB (3x2GB) set to my machine (corei7). I've now got 12GB RAM in windows 7 x64 professional.
In a crafty mod, and using VisualStudio 2010, when trying to allocate 2GB RAM my crafty compile gave the following error in HashProbe():
Unhandled exception at xxx adress,
0xC0000005: Access violation reading location 0x000000007dc23540.

Code: Select all

 for &#40;entry = 0; entry < 4; entry++, htable++) &#123;
    word1 = htable->word1;
    word2 = htable->word2 ^ word1; <-------- error here
    if &#40;word2 == temp_hashkey&#41;
      break;
 

I notice AllignedMalloc uses malloc.
What is the fix for this?
Maybe using VirtualMalloc ? If so what is the best way to implement Virtualalloc while keeping the hashtable alligned?

your comments will be much appreciated,
best regards,
Alvaro

Are you using a 64 bit compiler and a 64 bit operating system? Crafty has no "internal checks" for sanity of numbers like hash size. In looking at your note above, that makes no sense. How can "htable->word1" be perfectly OK, but "htable->word2" fail? In any case, here is Crafty running on a 12 gb machine with 8gb of hash:

White(1): hash=8192M
Warning-- xboard 'memory' option disabled
hash table memory = 8192M bytes (512M entries).
White(1): book off
book file disabled.
White(1): e4
time used: 27.04
time surplus 0.00 time limit 30.00 (+0.00) (3:00)
depth time score variation (1)
starting thread 1 2 3 4 5 6 7 8 9 10 11 <done>
16 0.48 0.20 1. ... Nc6 2. Nf3 e5 3. Nc3 Nf6 4.
Bc4 Bc5 5. O-O O-O 6. d3 d6 7. Bg5
h6 8. Be3 Be6 9. Qe2 Bxc4 10. Bxc5
dxc5 11. dxc4
16-> 0.59 0.20 1. ... Nc6 2. Nf3 e5 3. Nc3 Nf6 4.
Bc4 Bc5 5. O-O O-O 6. d3 d6 7. Bg5
h6 8. Be3 Be6 9. Qe2 Bxc4 10. Bxc5
dxc5 11. dxc4


So there is no reason I am aware of, EXCEPT for a version compiled with a 32 bit compiler, which will definitely break on hash sizes of 2gb and beyond because that turns into a negative number that will cause all sorts of problems.

Thanks Bob for your feedback.
I really must have made a boboo somewhere.
Yes the compiler is VS2010 and a x64 compile and Windows 7 x64.
However I'm now sure I must made something wrong.
I'll have to investigate.

thanks again,
Alvaro

1. which version of Crafty? A few in the past have had large hash issues in windows.

2. If this is a 23.4 (most current version) and it is failing, let me know. There is a different malloc() procedure for windows vs unix and there might well be some sort of casting issue that screws up. I force the hash pointer to start on a 64-byte memory address boundary for cache efficiency. That can break if it is done incorrectly...

Let me know which version and maybe it is something that needs fixing.

it's 23.4
but take notice I made some mods to be VS2010 friendly.

thanks,
Alvaro

Any changes related to the hash table pointer alignment stuff???

[Cardoso]

Hi,
I add another 6GB (3x2GB) set to my machine (corei7). I've now got 12GB RAM in windows 7 x64 professional.
In a crafty mod, and using VisualStudio 2010, when trying to allocate 2GB RAM my crafty compile gave the following error in HashProbe():
Unhandled exception at xxx adress,
0xC0000005: Access violation reading location 0x000000007dc23540.

Code: Select all

 for &#40;entry = 0; entry < 4; entry++, htable++) &#123;
    word1 = htable->word1;
    word2 = htable->word2 ^ word1; <-------- error here
    if &#40;word2 == temp_hashkey&#41;
      break;
 

I notice AllignedMalloc uses malloc.
What is the fix for this?
Maybe using VirtualMalloc ? If so what is the best way to implement Virtualalloc while keeping the hashtable alligned?

your comments will be much appreciated,
best regards,
Alvaro

Are you using a 64 bit compiler and a 64 bit operating system? Crafty has no "internal checks" for sanity of numbers like hash size. In looking at your note above, that makes no sense. How can "htable->word1" be perfectly OK, but "htable->word2" fail? In any case, here is Crafty running on a 12 gb machine with 8gb of hash:

White(1): hash=8192M
Warning-- xboard 'memory' option disabled
hash table memory = 8192M bytes (512M entries).
White(1): book off
book file disabled.
White(1): e4
time used: 27.04
time surplus 0.00 time limit 30.00 (+0.00) (3:00)
depth time score variation (1)
starting thread 1 2 3 4 5 6 7 8 9 10 11 <done>
16 0.48 0.20 1. ... Nc6 2. Nf3 e5 3. Nc3 Nf6 4.
Bc4 Bc5 5. O-O O-O 6. d3 d6 7. Bg5
h6 8. Be3 Be6 9. Qe2 Bxc4 10. Bxc5
dxc5 11. dxc4
16-> 0.59 0.20 1. ... Nc6 2. Nf3 e5 3. Nc3 Nf6 4.
Bc4 Bc5 5. O-O O-O 6. d3 d6 7. Bg5
h6 8. Be3 Be6 9. Qe2 Bxc4 10. Bxc5
dxc5 11. dxc4


So there is no reason I am aware of, EXCEPT for a version compiled with a 32 bit compiler, which will definitely break on hash sizes of 2gb and beyond because that turns into a negative number that will cause all sorts of problems.

Thanks Bob for your feedback.
I really must have made a boboo somewhere.
Yes the compiler is VS2010 and a x64 compile and Windows 7 x64.
However I'm now sure I must made something wrong.
I'll have to investigate.

thanks again,
Alvaro

1. which version of Crafty? A few in the past have had large hash issues in windows.

2. If this is a 23.4 (most current version) and it is failing, let me know. There is a different malloc() procedure for windows vs unix and there might well be some sort of casting issue that screws up. I force the hash pointer to start on a 64-byte memory address boundary for cache efficiency. That can break if it is done incorrectly...

Let me know which version and maybe it is something that needs fixing.

it's 23.4
but take notice I made some mods to be VS2010 friendly.

thanks,
Alvaro

Any changes related to the hash table pointer alignment stuff???

No, none related to that.
I just intended to play around with the "bad pruning" idea posted here:
http://www.talkchess.com/forum/viewtopi ... t=prunning
I didn't come to any conclusions.

Alvaro