en passant and hash key calculation

[Desperado]

I have been including a hash key component when a pawn moves forward two squares, regardless of whether or not there is a capture possible. As I think about it, this seems wrong, in that I eliminate a significant number of possible hash key matches and slow down my search.

doing the update like you have done so far is an absolutely correct way to handle this.
What you described as mistake is at most a missing optimization.

Making that optimization is worthwhile. After reading this useful thread, I now set an en passant square only if there is an opposing pawn that can capture it. My time for a nominal 10 ply search from the start went from 14.1 s to 11.6 s. A good result from 5 minutes' programming.

Robert P.

Nice speedup.

But i am interested in the reason for this. i dont think
it is (only) due a better ttb-hitrate. I would bet it is because you
drop much earlier into repetition detection.(especially if
you report repetition draw on count==1).
And because the start position include 16 pawns which can
be pushed double this leads to the nice speedup.

did you test for midgame position ?
what is the speedup when (just for testing) you lock the
repetition detection, to see if it is really the hashhit-rate
that is producing the gain?

regards Michael

[Sven]

I have been including a hash key component when a pawn moves forward two squares, regardless of whether or not there is a capture possible. As I think about it, this seems wrong, in that I eliminate a significant number of possible hash key matches and slow down my search.

doing the update like you have done so far is an absolutely correct way to handle this.
What you described as mistake is at most a missing optimization.

Making that optimization is worthwhile. After reading this useful thread, I now set an en passant square only if there is an opposing pawn that can capture it. My time for a nominal 10 ply search from the start went from 14.1 s to 11.6 s. A good result from 5 minutes' programming.

Robert P.

What you, as well as others, have described as an optimization is even more, it is also a bugfix that helps to avoid missing a couple of repetitions.

Sven

[Sven]

Hi Michael;

Now that you guys made me think about this I am not sure that the way that you (and I) are updating the zorbrist key is correct.

After.

1. e4 e6 2. e5 d5

I think that it is wrong to add in the e.p. zorbrist key after the double move because after

3. Nf3 Ne7 4. Ng1 Ng8

the hash key is the same as after the double move.

Instead.

1. e4 e6 2. e5 d5 3. Nf3

Now add in the e.p. zorbrist so that

3. ... Ne7 4. Ng1 Ng8

actually ends up with a different hash key than the position just after the double move.

If you do it like that, you definitely do it wrong!

Fully agreed.

The e.p. key should be added to the hash key when probing, but not to the differentially updated hash key. e.p. rights are a transient feature, and disappear after one ply. What you put in the differentially updated key stays there forever.

In micro-Max I have a differentially updated key where I incorporate from-square, two-square and captured victim in the usual way, but when probing, I do not use that key, but I add stm*epSquare first. The stm encoding in uMax is 8=white and 16=black, and the e.p. squares are square numbers on a 0x88 board, and range from 32-39 (white e.p. captures) and 80-87 (black e.p. captures), and 128 if there are no e.p. rights. So every combination of stm and e.p. will give me a different addition to the differential key.

This is an interesting proposal. However I would not say one "should" do it like this, it is just another correct way to do it, and it may depend on other properties of an engine which of the possible ways is the best. The branchless example of Michael H. (always XOR zobrist keys of old and new ep target into hash key) looks quite good, too, for me, although the code to manage the ep target itself misses the check for presence of adjacent enemy pawns.

Sven

[Sven]

...
in that I eliminate a significant number of possible hash key matches and slow down my search.

So, i dont see (at least at the moment) why this can eliminate significantly
the number of possible hash key matches.
None of the subtree positions (after double pawn push) is influenced by the zbrEpt, with the only exception of the "x-position" which has the
same piece-placement but simply another board status.
(writing this, i wonder what the optimization can be in the context of
hashhits ?)
@everyone: feel free to correct me

I think Fred was right with his statement, maybe you just misunderstood him? What he probably meant was that always setting the ep target after a double pawn step even if there is no adjacent enemy pawn creates an artificial (and also wrong) difference between the hash keys of

a) a position after a double pawn step where in fact no enemy pawn might capture en passant, and
b) a position with identical piece placement where the last move was not that double pawn step from a).

This is possible in case of repetition (i.e., b) occurs in a subtree of a) ) but also in case of transposition. Consider this stupid example:

a) 1.e4 c6 2.Nf3 c5 3.Nc3 e5 (ep target "wrongly" set to e6)
b) 1.e4 e6 2.Nf3 c5 3.Nc3 e5 (no ep target set)

Sven

[Desperado]

... although the code to manage the ep target itself misses the check for presence of adjacent enemy pawns.

Sven

is there a reason to identify them beside movegeneration issues ?

(at least using bitboards, you dont need to identify them seperately)

michael

[Sven]

... although the code to manage the ep target itself misses the check for presence of adjacent enemy pawns.

Sven

is there a reason to identify them beside movegeneration issues ?

(at least using bitboards, you dont need to identify them seperately)

michael

Well, it looks like that was the main content of this thread ...

Sven

[Sven]

I have been including a hash key component when a pawn moves forward two squares, regardless of whether or not there is a capture possible. As I think about it, this seems wrong, in that I eliminate a significant number of possible hash key matches and slow down my search.

doing the update like you have done so far is an absolutely correct way to handle this.
What you described as mistake is at most a missing optimization.

Making that optimization is worthwhile. After reading this useful thread, I now set an en passant square only if there is an opposing pawn that can capture it. My time for a nominal 10 ply search from the start went from 14.1 s to 11.6 s. A good result from 5 minutes' programming.

Robert P.

What you, as well as others, have described as an optimization is even more, it is also a bugfix that helps to avoid missing a couple of repetitions.

Sven

And also transpositions, as I pointed out in another post.
Sven

[Desperado]

... although the code to manage the ep target itself misses the check for presence of adjacent enemy pawns.

Sven

is there a reason to identify them beside movegeneration issues ?

(at least using bitboards, you dont need to identify them seperately)

michael

Well, it looks like that was the main content of this thread ...

Sven

of course !

...

a) 1.e4 c6 2.Nf3 c5 3.Nc3 e5 (ep target "wrongly" set to e6)
b) 1.e4 e6 2.Nf3 c5 3.Nc3 e5 (no ep target set)

...

a) 1.e4 c6 2.Nf3 c5 3.Nc3 e5 4.Nb1 Nf6 5.Nc3 Ng8 5.Nb1 (p1: is this a repetition ? - it s not with respect to ep status)
b) 1.e4 e6 2.Nf3 c5 3.Nc3 e5 4.Nb1 Nf6 5.Nc3 Ng8 5.Nb1 (p2: is this a repetition ? - it is ignoring the ep status)

So what i want to say is: to decide if it is a wrong/right
ep target we have to accord with all the other rules given.

all i can see at the moment is that we all (including myself) are
using our own interpretation. If for example the repetition
rule would give a clear answer if p1 is a repetition, we would
be able (by inverse logic) to define case a/b as correct/incorrect.

So, because i wanted the right defenition to solve that i just
phoned with a friend and chess arbiter. He acknowledged that
p1 is not a repetition.

That means that the ep-status has to be set in case a.
(well, how to handle internally is just another subject of course)

Michael

[Desperado]

i am not sure about my last post anymore !?



what i like to see is a clear definition (not interpretation) of what
causes to set the ep target.

thx michael

[Sven]

... although the code to manage the ep target itself misses the check for presence of adjacent enemy pawns.

Sven

is there a reason to identify them beside movegeneration issues ?

(at least using bitboards, you dont need to identify them seperately)

michael

Well, it looks like that was the main content of this thread ...

Sven

of course !

...

a) 1.e4 c6 2.Nf3 c5 3.Nc3 e5 (ep target "wrongly" set to e6)
b) 1.e4 e6 2.Nf3 c5 3.Nc3 e5 (no ep target set)

...

a) 1.e4 c6 2.Nf3 c5 3.Nc3 e5 4.Nb1 Nf6 5.Nc3 Ng8 5.Nb1 (p1: is this a repetition ? - it s not with respect to ep status)
b) 1.e4 e6 2.Nf3 c5 3.Nc3 e5 4.Nb1 Nf6 5.Nc3 Ng8 5.Nb1 (p2: is this a repetition ? - it is ignoring the ep status)

So what i want to say is: to decide if it is a wrong/right
ep target we have to accord with all the other rules given.

all i can see at the moment is that we all (including myself) are
using our own interpretation. If for example the repetition
rule would give a clear answer if p1 is a repetition, we would
be able (by inverse logic) to define case a/b as correct/incorrect.

So, because i wanted the right defenition to solve that i just
phoned with a friend and chess arbiter. He acknowledged that
p1 is not a repetition.

That means that the ep-status has to be set in case a.
(well, how to handle internally is just another subject of course)

Michael

I saw your next post while writing this answer and resuming from an interruption. So let me complete this one first ...

Your friend is right but you misunderstand the meaning of what he says.

- p1 is not a position where draw by the rules of chess can be claimed, since it is not a position that has occurred three times with identical piece placement and identical conditions (side to move, castling rights, ep target).

- However, p1 has occurred twice. It is identical to the position after "1.e4 c6 2.Nf3 c5 3.Nc3 e5 4.Nb1". So p1 is a repetition, and many engines return a draw score during search on repCount==1 for good reasons, so it is relevant.

- Neither p1, nor the position after 4.Nb1 that it repeats has the ep target square set, which is simply because in both cases the last move was not even a pawn move. So you are missing the point completely here. A pawn double step is a *required* precondition for setting the ep target square after a move.

- The same applies to p2. Here you also ask the question at the wrong point, i.e. after the move Nb1 instead of one ply earlier.

Sven