Stockfish Maximum Hash Size

[zullil]

In ucioption.cpp, the maximum for Hash is 2048 (MB). Can this be increased, or does other code assume that this Hash value is no more than 2048?

My reason for asking is this. In a typical search using 8 threads, Stockfish examines maybe 1024^3 nodes. If I assume that each hash entry is 16B, then I get 1024^3 * 16 / 1024^3 GB = 16 GB. If this calculation is correct, doesn't it suggest that a 2GB Hash is too small for me?

Thanks,

Louis

[mcostalba]

In ucioption.cpp, the maximum for Hash is 2048 (MB). Can this be increased, or does other code assume that this Hash value is no more than 2048?

My reason for asking is this. In a typical search using 8 threads, Stockfish examines maybe 1024^3 nodes. If I assume that each hash entry is 16B, then I get 1024^3 * 16 / 1024^3 GB = 16 GB. If this calculation is correct, doesn't it suggest that a 2GB Hash is too small for me?

Thanks,

Louis

Hi Louis,

I don't think I have clearly understood your numbers. But apart from that the main point is that the hash size is shared among the threads, so it doesn't mean that each thread has 2GB / 8 quantity of table, but each thread has the _whole_ table. Actually what happens is that one thread fills an entry that is then used by another one in a nice cooperation.

So I think we should not look at the TT table as a set of partitions each one dedicated to one thread.

And apart from this you cannot malloc more then 2GB of memory in one chunk (already tried), at least on a 32 bit OS, I am not sure on a 64 bit one.

[Milos]

And apart from this you cannot malloc more then 2GB of memory in one chunk (already tried), at least on a 32 bit OS, I am not sure on a 64 bit one.

If you are talking about windows this is not really true. You can allocate almost all 32bits (or what's free of 4 GB in case of 32b OS). For 64b there are no restrictions at all.
You just need to set properly the heap:
http://msdn.microsoft.com/en-us/library/f90ybzkh.aspx

[Sven]

In ucioption.cpp, the maximum for Hash is 2048 (MB). Can this be increased, or does other code assume that this Hash value is no more than 2048?

My reason for asking is this. In a typical search using 8 threads, Stockfish examines maybe 1024^3 nodes. If I assume that each hash entry is 16B, then I get 1024^3 * 16 / 1024^3 GB = 16 GB. If this calculation is correct, doesn't it suggest that a 2GB Hash is too small for me?

Thanks,

Louis

There is no 1:1 relation between number of nodes visited and number of "TT store" operations. For instance, a successful hash table lookup, null move pruning, or some conditions within qsearch may cause the search not to store anything in the TT. Iterative deepening also includes that many positions that have been searched before to a shallower depth are found in the TT and will now be searched deeper.

Therefore you get a much smaller number of "store" operations than you visit nodes.

Furthermore, I think the size of each hash entry in Stockfish may be smaller than 16B (I remember 12B but I am not sure, I did not look recently).

Finally, it is not necessarily the goal to store "all" positions of one single search within the hash table. It is not even clear whether an engine will always play stronger if it uses a bigger hash table, although this may be the case for Stockfish.

Sven

[Mincho Georgiev]

Marco, if i get that right, i suppose he meant that with 8 threads, the hash table is going to get filled (and therefore probably cleared if you are doing that as i do) a couple of times faster. I might misunderstood him of course.

[zullil]

Hi Marco,

The question I'm trying to answer is "What size hash would be best for my system (and might that number exceed 2 GB)?"

I typically run 8 threads, giving me about 8,000,000 nps. I am wondering if with Hash = 2048 the program encounters many positions that are not hashed, simply because there is not enough room to hash them. If so, wouldn't this slow down evaluation quite a bit?

I notice that when I search, the Hashfull number reported very quickly becomes 999 and then 1000, and then rolls over and quickly grows back to 999.

Sorry, if these questions are foolish or ill-posed; I'm not very knowledgeable about this topic.

Thanks.

[zullil]

Marco, if i get that right, i suppose he meant that with 8 threads, the hash table is going to get filled (and therefore probably cleared if you are doing that as i do) a couple of times faster. I might misunderstood him of course.

Yes, that's what I meant. Wouldn't it be better to have larger hash to avoid the filling and clearing?

Thanks.

[Milos]

Sorry, if these questions are foolish or ill-posed; I'm not very knowledgeable about this topic.

Your question is quite good actually and also well-known. What is the optimal size of the hash? It strongly depends on the engine itself. And the only way to tell it exactly is to measure performance with different hash sizes. This has already been done for various engines (don't know for Stockfish, would be interesting to find out), and the conclusion is that strength improvement starts to become quite small when hash values go above 2GB. Of course for faster hardware (more threads) more hash should be more beneficial but there is always a saturation point where you can't improve strength any more by adding more hash.

[jwes]

Sorry, if these questions are foolish or ill-posed; I'm not very knowledgeable about this topic.

Your question is quite good actually and also well-known. What is the optimal size of the hash? It strongly depends on the engine itself. And the only way to tell it exactly is to measure performance with different hash sizes. This has already been done for various engines (don't know for Stockfish, would be interesting to find out), and the conclusion is that strength improvement starts to become quite small when hash values go above 2GB. Of course for faster hardware (more threads) more hash should be more beneficial but there is always a saturation point where you can't improve strength any more by adding more hash.

It (of course) also depends on the time control, e.g. if you are doing overnight analysis of a position, you need as much hash as you have space for.

[Mincho Georgiev]

Marco, if i get that right, i suppose he meant that with 8 threads, the hash table is going to get filled (and therefore probably cleared if you are doing that as i do) a couple of times faster. I might misunderstood him of course.

Yes, that's what I meant. Wouldn't it be better to have larger hash to avoid the filling and clearing?

Thanks.

Only on theory. In practice, the difference will be very hard to measure and even notice. And I don't really know if Stockfish is clearing the hash after the last entry is saved. It was just a supposition. This is at least what i do. Anyway, you have a reasonable point in your question, but is very hard to measure the difference betw. 2048 hash and 4096 with confidence, while is easy to see it betw. 16MB and 2048MB. It just doesn't worth bothering (imo).