FreePascal and HashTables' size problem/anomaly

[JuLieN]

What, exactly, are you measuring and how? There are library buffers, a large program stack, lots of instructions, lots of linked in library modules, etc. Without knowing exactly what you are measuring and how you are measuring it, it is impossible to guess.

I'm not sure I understand the true meaning of your objection disguised into a question, Bob. But I'll answer anyway :

1) I measured three things :
- the total memory use of my engine (given both by Arena and any task manager)
- the size of a hash slot (16 bytes)
- the size of my hashtables

Doing that, I wanted to see how much non-hash memory my engine consumed, as Andrés Valverde and Olivier Deville used to nickname it "the RAM-Prédateur". And they were right : Pred 2.0 was using about 322MB of non-hash memory when using a 128MB big hashtable (or so I thought, as the story told). As their joke was cruel and true, serious measures had to be taken to put this memory-eater on diet (this is the problem, developing a chess engine with an iMac populated with 8GB of RAM: resources being plentiful one just doesn't care enough...).

So last week I reorganized lots of my data structures (especially my stupidly huge HistoryKillers array), and managed to shrink the memory use of all my arrays to about 6MB in total (yes, I computed their individual sizes and sum them up). I didn't add the individual variables, though.

Still, Prédateur was using 93MB of non-hash memory, which puzzled me as it didn't match the sums I had computed. That's where Joost gave me the correct answer : the compiler was by default just aligning each field of my hashtable to 64 bits, so the total size of a slot actually was 56 bytes, which explained the huge numbers. Problem corrected by telling the compiler to align them to 8 bits, with no noticeable speed penalty.

Now, what was the true meaning of your question? I know I am not a professional programmer, nor did I even took any programming class, so I'm sure I've been very approximative in my posts.

Funnily, Pred now uses 142MB of memory in all, for a 128 MB hash table. Which seems correct, considering that it also has a 8MB big pawnhash. 142-128-8 = 6MB. This matches my sums, and it shows that all the library buffers and such, you talked about, are of very marginal impact.

[bob]

What, exactly, are you measuring and how? There are library buffers, a large program stack, lots of instructions, lots of linked in library modules, etc. Without knowing exactly what you are measuring and how you are measuring it, it is impossible to guess.

I'm not sure I understand the true meaning of your objection disguised into a question, Bob. But I'll answer anyway :

It really wasn't an objection. Without knowing how you are measuring memory footprint, I can't begin to guess what is included.

1) I measured three things :
- the total memory use of my engine (given both by Arena and any task manager)
- the size of a hash slot (16 bytes)
- the size of my hashtables

Doing that, I wanted to see how much non-hash memory my engine consumed, as Andrés Valverde and Olivier Deville used to nickname it "the RAM-Prédateur". And they were right : Pred 2.0 was using about 322MB of non-hash memory when using a 128MB big hashtable (or so I thought, as the story told). As their joke was cruel and true, serious measures had to be taken to put this memory-eater on diet (this is the problem, developing a chess engine with an iMac populated with 8GB of RAM: resources being plentiful one just doesn't care enough...).

So last week I reorganized lots of my data structures (especially my stupidly huge HistoryKillers array), and managed to shrink the memory use of all my arrays to about 6MB in total (yes, I computed their individual sizes and sum them up). I didn't add the individual variables, though.

Still, Prédateur was using 93MB of non-hash memory, which puzzled me as it didn't match the sums I had computed. That's where Joost gave me the correct answer : the compiler was by default just aligning each field of my hashtable to 64 bits, so the total size of a slot actually was 56 bytes, which explained the huge numbers. Problem corrected by telling the compiler to align them to 8 bits, with no noticeable speed penalty.

Now, what was the true meaning of your question? I know I am not a professional programmer, nor did I even took any programming class, so I'm sure I've been very approximative in my posts.

You can encounter alignment issues making structure elements bigger. You can encounter alignment issues from the optimizer trying to align procedures on some boundary. You can find a huge stack you didn't know about. You can find non-shared library routines included that you were not counting. It is quite easy to jack up an executable's memory footprint without knowing it has happened. So that was why I asked what I did. remember, I have an operating systems background, and have been looking at program speed and size issues for 40+ years now. So I was trying to understand what numbers you were looking at so I could speculate.

Funnily, Pred now uses 142MB of memory in all, for a 128 MB hash table. Which seems correct, considering that it also has a 8MB big pawnhash. 142-128-8 = 6MB. This matches my sums, and it shows that all the library buffers and such, you talked about, are of very marginal impact.