How often do engines clear their hash memory

[Onegin]

I've noticed that some engines (like Komodo 10 and Komodo 9) do NOT clear their hash memory between moves and accumulate hash until it gets to 99.999% or 100%, whereas others (like Stockfish 8 and Stockfish 7) clear their memory between moves. What is the reasoning for this behavior?

[Dann Corbit]

I've noticed that some engines (like Komodo 10 and Komodo 9) do NOT clear their hash memory between moves and accumulate hash until it gets to 99.999% or 100%, whereas others (like Stockfish 8 and Stockfish 7) clear their memory between moves. What is the reasoning for this behavior?

I guess that Komodo has a better replacement scheme.

In general, if I am analyzing a game or a pile of EPD records that are related to one another, I do not want the engine to clear hash between moves.

But if it finds a position that is not found in the hash, then it would be sensible to clear it.

I will get double the depth of analysis if I analyze closely related positions without clearing the hash. But if the positions are not related and I do not clear the hash, then the analysis will be hampered because the hash table is full of the wrong stuff.

It is even more important if I am backsolving, by analyzing the game backwards.

[mjlef]

I've noticed that some engines (like Komodo 10 and Komodo 9) do NOT clear their hash memory between moves and accumulate hash until it gets to 99.999% or 100%, whereas others (like Stockfish 8 and Stockfish 7) clear their memory between moves. What is the reasoning for this behavior?

All programs I have seen just keep replacing hash entries. Komodo only clears the hash used counter when Clear Hash is selected. Other programs reset it with each new search. The UCI spec does not say what is correct, but since I have grown tired of explaining this I am likely to change this in a future version. Don programmed it this way and I have just left it. It is really only useful in helping select the Hash size when you choose a different time control or number of Threads.

Hash tables are never really "full". They just replace existing entries (aging the table). Programs would play much worse if they cleared the Hash table between moves. But it is best to be able to hold most of a given search in memory, to prevent researching (and for move ordering).

[Onegin]

Thanks for your explanation. I guess it comes with "fame and fortune", the necessity to answer same questions over and over again... If you still have a second...
It looks from my inexperienced point of view that keeping hash is a better/more efficient approach. But then why do latest versions of Stockfish (7 and 8) clear hash between moves?

[Dann Corbit]

I've noticed that some engines (like Komodo 10 and Komodo 9) do NOT clear their hash memory between moves and accumulate hash until it gets to 99.999% or 100%, whereas others (like Stockfish 8 and Stockfish 7) clear their memory between moves. What is the reasoning for this behavior?

All programs I have seen just keep replacing hash entries. Komodo only clears the hash used counter when Clear Hash is selected. Other programs reset it with each new search. The UCI spec does not say what is correct, but since I have grown tired of explaining this I am likely to change this in a future version. Don programmed it this way and I have just left it. It is really only useful in helping select the Hash size when you choose a different time control or number of Threads.

Hash tables are never really "full". They just replace existing entries (aging the table). Programs would play much worse if they cleared the Hash table between moves. But it is best to be able to hold most of a given search in memory, to prevent researching (and for move ordering).

Pretty please, down on my knees, don't simply remove it.
If you make it a UCI option with false as the default that is fine.

It literally gives one free ply when analyzing consecutive positions from a game or from an opening being studied.

It is like getting twice as much time for free.

It was me who asked Don for this.

[Dann Corbit]

Thanks for your explanation. I guess it comes with "fame and fortune", the necessity to answer same questions over and over again... If you still have a second...
It looks from my inexperienced point of view that keeping hash is a better/more efficient approach. But then why do latest versions of Stockfish (7 and 8) clear hash between moves?

It was Forest Gump who said, "Stupid is as stupid does."

[Dann Corbit]

Thanks for your explanation. I guess it comes with "fame and fortune", the necessity to answer same questions over and over again... If you still have a second...
It looks from my inexperienced point of view that keeping hash is a better/more efficient approach. But then why do latest versions of Stockfish (7 and 8) clear hash between moves?

It was Forest Gump who said, "Stupid is as stupid does."

It does make sense under the following scenario:
1. You have a substandard replacement scheme
2. You are analyzing a group of unrelated positions.

Clearing the hash will make the analysis better in that case [processing a list of unrelated positions].

[Evert]

But if the positions are not related and I do not clear the hash, then the analysis will be hampered because the hash table is full of the wrong stuff.

This seems surprising: most engines will always store something at the end of the search, in which case it should not matter if there is unrelated junk in the hash table, since it will be replaced by something useful anyway.

[Dann Corbit]

But if the positions are not related and I do not clear the hash, then the analysis will be hampered because the hash table is full of the wrong stuff.

This seems surprising: most engines will always store something at the end of the search, in which case it should not matter if there is unrelated junk in the hash table, since it will be replaced by something useful anyway.

It only matters if you do not have a really good replacement strategy.

[hgm]

Unused hash is useless hash...

The only reason I can see for clearing the table before every move is reproducibility of bugs; on a fully debugged engine there is no need to do that. Even clearing before a new game isn't really useful. (Unless you changed variant...)

I have had a lot of trouble finding a replecement scheme that works satisfactorily with back-solving. Especially in the case where the main line forks to two branches, both to deep to recognize their value from the forking point. Then I have to back-solve from one branch end-point to the fork, and then from the other, to get them both correct in the fork (in particular, both bad). Very often the back solving of one branch uses so much hash that the score of the other branch has been overwritten, and vice versa.

In particular schemes that use aging by search number make the engine practically useless, when they count every move or undo during the analysis session as a new search. Because that would immediately declare the positions of branch A fair game for replacement when you enter branch B.

OTOH, if you consider the entire analysis session one search, the depth-preferred part of the table fills up completely with the deepest entries of all searches combined, leaving no room for the current search in a long session. The search becomes very sluggish then. So my currently favored replacement scheme uses a small part of the table for depth-preferred + aging, (where during a analysis session aging is suppressed), and uses the rest of the table with equidistributed draft, (where even un-aged deep entries can be replaced by shallower ones when their draft occupies more than its fair share of the table).