fine #70 revisited

[jwes]

looking again at the pv it seems that the pv is wrong

1. Kb1 Kb7 2. Kc1 Kb8 3. Kc2 Kb7 4. Kc3 Kb6 5. Kd2 Ka6 6. Kd3 Kb6 7. Ke2 Kc7 8. Kd1

It seems that 8.Kd1 is a drawing move when 8.Kf3 is winning.

This is hash PV grafting. This is a problem that remains, because when I get a hash hit, I can graft the existing PV from that hit, but I have no way to check to see if this path would include a repetition, since the hash information can not possibly tell me that... Apparently, if there were no repetition/50-move rules in force, Kd1 would be ok... If you set it up from that position (prior to Kd1) with this:

8/2k5/3p4/p2P1p2/P2P1P2/8/4K3/8 w - -

Which gets rid of the repetition list, the position is just as winnable as it always was...

Nothing to do to fix that except to completely break hashing by including path information in the hash signature...

When you get a hash hit with a non-drawish score, you could check the pv cache for repetitions. This would be good for analysis and might be good for games since it could avoid a 3-fold rep.

Doesn't do any good. What should I do then, just ignore the PV? and use the hash score? This is _very_ rare, in general. Obviously for positions like Fine where you can hit 50 plies in no time it becomes an issue...

Just because I see a draw in the path does not mean that root move is a draw. Obviously Kb1 wins. And even though the PV (due to grafting) has a repetition, the move still wins... I am not sure how you could learn anything from this...

Just don't cut off and keep searching. Hash cutoffs are rare enough that the cost should be very small.

[bob]

looking again at the pv it seems that the pv is wrong

1. Kb1 Kb7 2. Kc1 Kb8 3. Kc2 Kb7 4. Kc3 Kb6 5. Kd2 Ka6 6. Kd3 Kb6 7. Ke2 Kc7 8. Kd1

It seems that 8.Kd1 is a drawing move when 8.Kf3 is winning.

This is hash PV grafting. This is a problem that remains, because when I get a hash hit, I can graft the existing PV from that hit, but I have no way to check to see if this path would include a repetition, since the hash information can not possibly tell me that... Apparently, if there were no repetition/50-move rules in force, Kd1 would be ok... If you set it up from that position (prior to Kd1) with this:

8/2k5/3p4/p2P1p2/P2P1P2/8/4K3/8 w - -

Which gets rid of the repetition list, the position is just as winnable as it always was...

Nothing to do to fix that except to completely break hashing by including path information in the hash signature...

When you get a hash hit with a non-drawish score, you could check the pv cache for repetitions. This would be good for analysis and might be good for games since it could avoid a 3-fold rep.

Doesn't do any good. What should I do then, just ignore the PV? and use the hash score? This is _very_ rare, in general. Obviously for positions like Fine where you can hit 50 plies in no time it becomes an issue...

Just because I see a draw in the path does not mean that root move is a draw. Obviously Kb1 wins. And even though the PV (due to grafting) has a repetition, the move still wins... I am not sure how you could learn anything from this...

Just don't cut off and keep searching. Hash cutoffs are rare enough that the cost should be very small.

You should try searching fine 70 without hash hits. What do you do about the hash fail highs that don't consider reps? Hash fail lows? Hash exacts? This is not fixable in this way. All the current code does is show the PV that goes with the exact hash hit. There is only one solution to fix the problem, and it will effectively disable hashing entirely, which is not even remotely acceptable...

[jwes]

looking again at the pv it seems that the pv is wrong

1. Kb1 Kb7 2. Kc1 Kb8 3. Kc2 Kb7 4. Kc3 Kb6 5. Kd2 Ka6 6. Kd3 Kb6 7. Ke2 Kc7 8. Kd1

It seems that 8.Kd1 is a drawing move when 8.Kf3 is winning.

This is hash PV grafting. This is a problem that remains, because when I get a hash hit, I can graft the existing PV from that hit, but I have no way to check to see if this path would include a repetition, since the hash information can not possibly tell me that... Apparently, if there were no repetition/50-move rules in force, Kd1 would be ok... If you set it up from that position (prior to Kd1) with this:

8/2k5/3p4/p2P1p2/P2P1P2/8/4K3/8 w - -

Which gets rid of the repetition list, the position is just as winnable as it always was...

Nothing to do to fix that except to completely break hashing by including path information in the hash signature...

When you get a hash hit with a non-drawish score, you could check the pv cache for repetitions. This would be good for analysis and might be good for games since it could avoid a 3-fold rep.

Doesn't do any good. What should I do then, just ignore the PV? and use the hash score? This is _very_ rare, in general. Obviously for positions like Fine where you can hit 50 plies in no time it becomes an issue...

Just because I see a draw in the path does not mean that root move is a draw. Obviously Kb1 wins. And even though the PV (due to grafting) has a repetition, the move still wins... I am not sure how you could learn anything from this...

Just don't cut off and keep searching. Hash cutoffs are rare enough that the cost should be very small.

You should try searching fine 70 without hash hits. What do you do about the hash fail highs that don't consider reps? Hash fail lows? Hash exacts? This is not fixable in this way. All the current code does is show the PV that goes with the exact hash hit. There is only one solution to fix the problem, and it will effectively disable hashing entirely, which is not even remotely acceptable...

If it is an exact hash hit that would cause a cutoff, then check the stored pv for a repetition that spans the current node, e.g. a position that occurred before the current node also occurs in the stored pv, and if there is one, do not cut off.

[bob]

looking again at the pv it seems that the pv is wrong

1. Kb1 Kb7 2. Kc1 Kb8 3. Kc2 Kb7 4. Kc3 Kb6 5. Kd2 Ka6 6. Kd3 Kb6 7. Ke2 Kc7 8. Kd1

It seems that 8.Kd1 is a drawing move when 8.Kf3 is winning.

This is hash PV grafting. This is a problem that remains, because when I get a hash hit, I can graft the existing PV from that hit, but I have no way to check to see if this path would include a repetition, since the hash information can not possibly tell me that... Apparently, if there were no repetition/50-move rules in force, Kd1 would be ok... If you set it up from that position (prior to Kd1) with this:

8/2k5/3p4/p2P1p2/P2P1P2/8/4K3/8 w - -

Which gets rid of the repetition list, the position is just as winnable as it always was...

Nothing to do to fix that except to completely break hashing by including path information in the hash signature...

When you get a hash hit with a non-drawish score, you could check the pv cache for repetitions. This would be good for analysis and might be good for games since it could avoid a 3-fold rep.

Doesn't do any good. What should I do then, just ignore the PV? and use the hash score? This is _very_ rare, in general. Obviously for positions like Fine where you can hit 50 plies in no time it becomes an issue...

Just because I see a draw in the path does not mean that root move is a draw. Obviously Kb1 wins. And even though the PV (due to grafting) has a repetition, the move still wins... I am not sure how you could learn anything from this...

Just don't cut off and keep searching. Hash cutoffs are rare enough that the cost should be very small.

You should try searching fine 70 without hash hits. What do you do about the hash fail highs that don't consider reps? Hash fail lows? Hash exacts? This is not fixable in this way. All the current code does is show the PV that goes with the exact hash hit. There is only one solution to fix the problem, and it will effectively disable hashing entirely, which is not even remotely acceptable...

If it is an exact hash hit that would cause a cutoff, then check the stored pv for a repetition that spans the current node, e.g. a position that occurred before the current node also occurs in the stored pv, and if there is one, do not cut off.

What does that do? Exact hashes are very rare compared to bounds hashes. Yet the bounds hashes influence the PV just as much since they cause branches to be pruned away as winning or losing, when in fact they could be drawing due to repetition or 50 move issues...

If you ignore exact hashes but accept the others, you are inviting a ton of search inconsistencies, not to mention losing some Elo here and there...

[michiguel]

looking again at the pv it seems that the pv is wrong

1. Kb1 Kb7 2. Kc1 Kb8 3. Kc2 Kb7 4. Kc3 Kb6 5. Kd2 Ka6 6. Kd3 Kb6 7. Ke2 Kc7 8. Kd1

It seems that 8.Kd1 is a drawing move when 8.Kf3 is winning.

This is hash PV grafting. This is a problem that remains, because when I get a hash hit, I can graft the existing PV from that hit, but I have no way to check to see if this path would include a repetition, since the hash information can not possibly tell me that... Apparently, if there were no repetition/50-move rules in force, Kd1 would be ok... If you set it up from that position (prior to Kd1) with this:

8/2k5/3p4/p2P1p2/P2P1P2/8/4K3/8 w - -

Which gets rid of the repetition list, the position is just as winnable as it always was...

Nothing to do to fix that except to completely break hashing by including path information in the hash signature...

When you get a hash hit with a non-drawish score, you could check the pv cache for repetitions. This would be good for analysis and might be good for games since it could avoid a 3-fold rep.

Doesn't do any good. What should I do then, just ignore the PV? and use the hash score? This is _very_ rare, in general. Obviously for positions like Fine where you can hit 50 plies in no time it becomes an issue...

Just because I see a draw in the path does not mean that root move is a draw. Obviously Kb1 wins. And even though the PV (due to grafting) has a repetition, the move still wins... I am not sure how you could learn anything from this...

Just don't cut off and keep searching. Hash cutoffs are rare enough that the cost should be very small.

That seems to be too drastic. We discussed this before: Not cutting in PV is enough to guarantee a sane PV and it does not take too much extra time. Bob does not like it because it loses 3 ELO or so according to him,l but even if that is true it should be an option to be used in analyze mode.

Miguel

[bob]

looking again at the pv it seems that the pv is wrong

1. Kb1 Kb7 2. Kc1 Kb8 3. Kc2 Kb7 4. Kc3 Kb6 5. Kd2 Ka6 6. Kd3 Kb6 7. Ke2 Kc7 8. Kd1

It seems that 8.Kd1 is a drawing move when 8.Kf3 is winning.

This is hash PV grafting. This is a problem that remains, because when I get a hash hit, I can graft the existing PV from that hit, but I have no way to check to see if this path would include a repetition, since the hash information can not possibly tell me that... Apparently, if there were no repetition/50-move rules in force, Kd1 would be ok... If you set it up from that position (prior to Kd1) with this:

8/2k5/3p4/p2P1p2/P2P1P2/8/4K3/8 w - -

Which gets rid of the repetition list, the position is just as winnable as it always was...

Nothing to do to fix that except to completely break hashing by including path information in the hash signature...

When you get a hash hit with a non-drawish score, you could check the pv cache for repetitions. This would be good for analysis and might be good for games since it could avoid a 3-fold rep.

Doesn't do any good. What should I do then, just ignore the PV? and use the hash score? This is _very_ rare, in general. Obviously for positions like Fine where you can hit 50 plies in no time it becomes an issue...

Just because I see a draw in the path does not mean that root move is a draw. Obviously Kb1 wins. And even though the PV (due to grafting) has a repetition, the move still wins... I am not sure how you could learn anything from this...

Just don't cut off and keep searching. Hash cutoffs are rare enough that the cost should be very small.

That seems to be too drastic. We discussed this before: Not cutting in PV is enough to guarantee a sane PV and it does not take too much extra time. Bob does not like it because it loses 3 ELO or so according to him,l but even if that is true it should be an option to be used in analyze mode.

Miguel

I think you are mixing apples and oranges. this is an EXACT hit where I graft the PV I stored along with the exact entry. But that is a tiny part of the _real_ problem. What about all the LOWER/UPPER positions you reach that suggest the position is winning or losing, when, in fact, it would be drawn by rep or 50 move rule? Getting rid of those would completely kill performance and wouldn't be just a few elo difference.

This is a systemic issue. I only wanted to solve the problem of "short PVs" which I have come close to doing. I can't guarantee there is never a short one, because I store the path in a different hash table and it, too, is subject to overwriting. But the current approach does work very well. It was not intended to solve the "there's a draw along the path but the hash table doesn't see it" problem at all. That is a much bigger problem and just not using EXACT entries in the PV only addresses one tiny problem, not the larger problem where the real drawing issues are.

[michiguel]

looking again at the pv it seems that the pv is wrong

1. Kb1 Kb7 2. Kc1 Kb8 3. Kc2 Kb7 4. Kc3 Kb6 5. Kd2 Ka6 6. Kd3 Kb6 7. Ke2 Kc7 8. Kd1

It seems that 8.Kd1 is a drawing move when 8.Kf3 is winning.

This is hash PV grafting. This is a problem that remains, because when I get a hash hit, I can graft the existing PV from that hit, but I have no way to check to see if this path would include a repetition, since the hash information can not possibly tell me that... Apparently, if there were no repetition/50-move rules in force, Kd1 would be ok... If you set it up from that position (prior to Kd1) with this:

8/2k5/3p4/p2P1p2/P2P1P2/8/4K3/8 w - -

Which gets rid of the repetition list, the position is just as winnable as it always was...

Nothing to do to fix that except to completely break hashing by including path information in the hash signature...

When you get a hash hit with a non-drawish score, you could check the pv cache for repetitions. This would be good for analysis and might be good for games since it could avoid a 3-fold rep.

Doesn't do any good. What should I do then, just ignore the PV? and use the hash score? This is _very_ rare, in general. Obviously for positions like Fine where you can hit 50 plies in no time it becomes an issue...

Just because I see a draw in the path does not mean that root move is a draw. Obviously Kb1 wins. And even though the PV (due to grafting) has a repetition, the move still wins... I am not sure how you could learn anything from this...

Just don't cut off and keep searching. Hash cutoffs are rare enough that the cost should be very small.

That seems to be too drastic. We discussed this before: Not cutting in PV is enough to guarantee a sane PV and it does not take too much extra time. Bob does not like it because it loses 3 ELO or so according to him,l but even if that is true it should be an option to be used in analyze mode.

Miguel

I think you are mixing apples and oranges. this is an EXACT hit where I graft the PV I stored along with the exact entry. But that is a tiny part of the _real_ problem. What about all the LOWER/UPPER positions you reach that suggest the position is winning or losing, when, in fact, it would be drawn by rep or 50 move rule? Getting rid of those would completely kill performance and wouldn't be just a few elo difference.

This is a systemic issue. I only wanted to solve the problem of "short PVs" which I have come close to doing. I can't guarantee there is never a short one, because I store the path in a different hash table and it, too, is subject to overwriting. But the current approach does work very well. It was not intended to solve the "there's a draw along the path but the hash table doesn't see it" problem at all. That is a much bigger problem and just not using EXACT entries in the PV only addresses one tiny problem, not the larger problem where the real drawing issues are.

If you do not cut in the PV, you always have a sane PV. That is what I said and I am not mixing fruits I am digressing, which is not the same.

Miguel

[bob]

looking again at the pv it seems that the pv is wrong

1. Kb1 Kb7 2. Kc1 Kb8 3. Kc2 Kb7 4. Kc3 Kb6 5. Kd2 Ka6 6. Kd3 Kb6 7. Ke2 Kc7 8. Kd1

It seems that 8.Kd1 is a drawing move when 8.Kf3 is winning.

This is hash PV grafting. This is a problem that remains, because when I get a hash hit, I can graft the existing PV from that hit, but I have no way to check to see if this path would include a repetition, since the hash information can not possibly tell me that... Apparently, if there were no repetition/50-move rules in force, Kd1 would be ok... If you set it up from that position (prior to Kd1) with this:

8/2k5/3p4/p2P1p2/P2P1P2/8/4K3/8 w - -

Which gets rid of the repetition list, the position is just as winnable as it always was...

Nothing to do to fix that except to completely break hashing by including path information in the hash signature...

When you get a hash hit with a non-drawish score, you could check the pv cache for repetitions. This would be good for analysis and might be good for games since it could avoid a 3-fold rep.

Doesn't do any good. What should I do then, just ignore the PV? and use the hash score? This is _very_ rare, in general. Obviously for positions like Fine where you can hit 50 plies in no time it becomes an issue...

Just because I see a draw in the path does not mean that root move is a draw. Obviously Kb1 wins. And even though the PV (due to grafting) has a repetition, the move still wins... I am not sure how you could learn anything from this...

Just don't cut off and keep searching. Hash cutoffs are rare enough that the cost should be very small.

That seems to be too drastic. We discussed this before: Not cutting in PV is enough to guarantee a sane PV and it does not take too much extra time. Bob does not like it because it loses 3 ELO or so according to him,l but even if that is true it should be an option to be used in analyze mode.

Miguel

I think you are mixing apples and oranges. this is an EXACT hit where I graft the PV I stored along with the exact entry. But that is a tiny part of the _real_ problem. What about all the LOWER/UPPER positions you reach that suggest the position is winning or losing, when, in fact, it would be drawn by rep or 50 move rule? Getting rid of those would completely kill performance and wouldn't be just a few elo difference.

This is a systemic issue. I only wanted to solve the problem of "short PVs" which I have come close to doing. I can't guarantee there is never a short one, because I store the path in a different hash table and it, too, is subject to overwriting. But the current approach does work very well. It was not intended to solve the "there's a draw along the path but the hash table doesn't see it" problem at all. That is a much bigger problem and just not using EXACT entries in the PV only addresses one tiny problem, not the larger problem where the real drawing issues are.

If you do not cut in the PV, you always have a sane PV. That is what I said and I am not mixing fruits I am digressing, which is not the same.

Miguel

Maybe the PV will be "sane" or maybe it won't... Don't forget that the hash LOWER/UPPER entries can exert great influence over the PV. So that the program might well not show an optimal PV there either...

I see no reason to drop 3 Elo (eliminating the exact hits in PV) and still continue to produce a PV that may well not represent the best (or even a good) line of play because of the draw errors in the branches that influence the PV without actually being a part of it...