On-line engine blitz tourney November

[Volker Annuss]

I have added pawn hash since the last tournament. I was wondering what would be a good memory allocation ratio for hash/pawn_hash. Assuming hash and pawn hash use the same size of entry, would a ratio of 10:1 be good?

A pawn hash table should never change your search tree. It just speeds things up. So you can easily test which size of your pawn hash table is optimal by looking at the time to depth for some positions.

Don't test with the start position because this will show a larger optimal size.

My pawn hash table has a fixed size of 512 entries per thread. Larger sizes do not make much difference for me. But my pawn hash table is different. The entries are about 100 bytes. They hold information about score, open files, passers, pawn shield, weak pawns and more to be used in other parts of the evaluation, that involve other pieces too.

[flok]

I read the wiki on the pawn hash table but it is not entirely clear to me. What is it that one hashes? Do you have e.g. a board-setup with only the pawns and then calc a zobrist of that?

[Sven]

I read the wiki on the pawn hash table but it is not entirely clear to me. What is it that one hashes? Do you have e.g. a board-setup with only the pawns and then calc a zobrist of that?

There is a "pure" form of pawn hash, and there are also more sophisticated forms. The latter covers more information than that about pawns, e.g. kings and related scoring data (like pawn shields) may be included. I'll explain the former only. Goal is to store the pawn evaluation part of the static evaluation, excluding everything that depends on other pieces (so passed pawn eval may be in or out, depending on how you have defined it). You maintain a second hash key built from pawn locations only (I think ep square and side to move can be left out since this is not for search but for static eval). The most basic pawn hash entry will include the hash key (or the non-index part of it, as usual) and a score. But usually you will include more data that you want to use for other parts of the evaluation (like open/semi-open files, passed pawn locations) and that will be expensive to recalculate.

Then the pawn evaluation will work like this:
- Lookup current position in pawn hash
- found? => use data from pawn hash
- else => calculate pawn evaluation data and store it in pawn hash

The pawn hash key (for the "pure" form) only changes on pawn moves and moves that capture a pawn. It should be updated incrementally.

[D Sceviour]

Here is a method for the long word element of a pawn hash entry, in my order of importance:

(1) A pawn hash key that is calculated with a zobrist. This is a different hash key than regular hash. The hash key also includes different values for en passant square when en passant is available.

(2) A pawn score for the pawn structure. This is an essential and a minimal structure entry. One could stop here, but further additions are use useful in multiples of 2 - 1,2,4,8,16 etc. for the classical method of masked probing code. This is a typical pawn hash entry that can be compared for hash/pawn hash ratio. Two entry sizes must follow for the pawn hash table if more entries are wanted:

(3) A pawn bitboard of the white pawns.

(4) A pawn bitboard of the black pawns.

Now four entry sizes must follow for the pawn hash table if more entries are wanted:

(5) The white pawn attack board. These can be created with two logical shifts <<7 and >>9 of the pawn bitboard.

(6) The black pawn attack board.

(7) A passed pawn bitboard of all the passed pawns. The individual color of the passed pawn can be extracted. This is very useful in different parts of the search and evaluation.

(8) Any other entry you may want here. Pawn defects are a good one that takes a lot of time to calculate.

Eight more entries must follow now if you can think of anything useful: doubled pawns, backward pawns, isolated pawns, pawn file sets. The entry is growing quite large at this point and more memory is required to store with each doubling of the entry size.

However, the original question has not been answered. What is the best memory allocation ratio for hash to pawn hash? When running multiple tests, a fixed amount of memory is usually available for each engine - for example 256M. Hash is more important than pawn hash. Would a ratio of 230M hash + 26M pawn hash = 256M total hash memory be satisfactory? Volker Annuss has suggested above testing the engine with node counts. I may get around to that, but I was wondering what ratios anyone else may suggest. Of course, for the ICS tournament one can allocate as much unnecessary memory as available.

[Daniel Anulliero]

I've implemented pawn hash in Isa some months ago , but it produced some diférents move sometimes wich I think is not very normal
I've posted about this issue here and the main answer was an hash bug ..
What do you think ?

[D Sceviour]

I've implemented pawn hash in Isa some months ago , but it produced some diférents move sometimes wich I think is not very normal
I've posted about this issue here and the main answer was an hash bug ..
What do you think ?

It is hard to say without knowing more about your program. The pawn hash should simply speed up the engine. It should not produce a different score or a different move. When making a major change like pawn hash, I like to compare two identical versions - one without the pawn hash table, and one with the new pawn hash. Then I edit until the score, moves produced, and node counts are the same. What might produce instant errors is not considering the en passant status correctly.

[Sven]

What might produce instant errors is not considering the en passant status correctly.

Why do you think that the en passant status is relevant for the pawn hash table? In my opinion it should be irrelevant there, unless you have an evaluation component that assigns different scores with vs. without en passant capability.

[D Sceviour]

What might produce instant errors is not considering the en passant status correctly.

Why do you think that the en passant status is relevant for the pawn hash table? In my opinion it should be irrelevant there, unless you have an evaluation component that assigns different scores with vs. without en passant capability.

Exactly. The following similar position could have two different pawn hash keys and scores - one for en passant available (white wins), and one for en passant not available (black wins). The difference in the FEN position is defined by the en passant square at b6.
[d]6k1/8/p7/Pp6/8/8/5K2/8 w - b6 0 1

[Daniel Anulliero]

I've implemented pawn hash in Isa some months ago , but it produced some diférents move sometimes wich I think is not very normal
I've posted about this issue here and the main answer was an hash bug ..
What do you think ?

It is hard to say without knowing more about your program. The pawn hash should simply speed up the engine. It should not produce a different score or a different move. When making a major change like pawn hash, I like to compare two identical versions - one without the pawn hash table, and one with the new pawn hash. Then I edit until the score, moves produced, and node counts are the same. What might produce instant errors is not considering the en passant status correctly.

Thank you for the answer
One day I'll have a look at it again

[Volker Annuss]

However, the original question has not been answered. What is the best memory allocation ratio for hash to pawn hash? When running multiple tests, a fixed amount of memory is usually available for each engine - for example 256M. Hash is more important than pawn hash. Would a ratio of 230M hash + 26M pawn hash = 256M total hash memory be satisfactory? Volker Annuss has suggested above testing the engine with node counts. I may get around to that, but I was wondering what ratios anyone else may suggest. Of course, for the ICS tournament one can allocate as much unnecessary memory as available.

For my engines it is sufficient to use a small fixed size for the pawn hash. Small enough to ignore its memory usage.

If this is not an option to you and you don't want to exceed the configured hash size, make your main hash a little smaller. Although normally a power of 2 is chosen for the main hash size, there is no need for it.

You can use any number of hash buckets with

Code: Select all

bucket_index = ((zobrist_key >> 32) * number_of_buckets) >> 32;

which is not much slower than the usual way.