Transposition table pruning

[zamar]

I don't understand the last point. Why would TT mate-scores in the PV need different handling than in non-PV nodes?

Hmm... We had problems with this behaviour when qsearch PV-nodes accidentally probed TT. But maybe if probing is done systematically everywhere, this is not a problem. I'm not sure about this Thanks for correction!

[zamar]

The TT can hide a draw because it doesn't store path information. But this is true of the 50-move rule as well. I certainly do not any justification for not allowing cutoffs from hash along the PV. It is just a hack fix that actually loses significant information. This is critical for finding fine 70 early, as an example.

Interesting. I'll add this to my testing list. Perhaps I need to study Crafty how you can cope with the repetitions and 50-move rule

[bob]

Values from TT are not used in PV, because:

* Extra cost is small (TT values can be used for non-PV child nodes).
* rep. draws are problemtic (as hmg pointed out)
* fifty move rule detection is problematic
* delivering mate is problematic when TT shows matescores everywhere and would need special handling.

I do not use TT in PV because I like to have a full PV. But, I never had any of the other problems. Why would you have specific problems with mates scores and draws?

Miguel

The TT can hide a draw because it doesn't store path information. But this is true of the 50-move rule as well. I certainly do not any justification for not allowing cutoffs from hash along the PV. It is just a hack fix that actually loses significant information. This is critical for finding fine 70 early, as an example.

I think you missed the word "specific" in my question. What you say is right but not specific for the PV.

Miguel

I was actually just responding to the "general case". I see no reason to treat PV vs non-PV nodes any differently. That is a bad misconception. Why would you want to search the PV one way, and then search the other moves another way? Isn't what is good for the PV good for the rest? Don't you want the rest to have an opportunity to fail high and replace the original PV? This is one of those ideas that simply makes absolutely no sense in any type of alpha/beta minimax tree search application. You can treat moves differently if you wish. But if it gets done differently based on PV vs non-PV that too is flawed.

I doubt we disagree, I just wasn't directly addressing only the PV case, which you were apparently responding to.

[bob]

The TT can hide a draw because it doesn't store path information. But this is true of the 50-move rule as well. I certainly do not any justification for not allowing cutoffs from hash along the PV. It is just a hack fix that actually loses significant information. This is critical for finding fine 70 early, as an example.

Interesting. I'll add this to my testing list. Perhaps I need to study Crafty how you can cope with the repetitions and 50-move rule

All I do is if I near the 50-move rule case, I clear the hash scores (mark each entry as "score is invalid" but I do not clear entries as I want best moves).

There is really nothing you can do to solve either of these two problems unless you choose to incorporate path information into the hash signature, and I'm not willing to do that as it wlil effectively wipe out the hash cutoffs completely. Ugly idea.

You can either toss out hashing and solve the problem completely, or else cripple things and still not solve the problem completely. I'd rather have the hash hits cause cutoffs and accept the errors dealing with reps/50-moves rather than vice-versa. I ran this test last year and reported the results. I don't remember how much this hurt elo overall, so I should probably just run it again disabling early hash exits when alpha != beta - 1...

[metax]

I was actually just responding to the "general case". I see no reason to treat PV vs non-PV nodes any differently. That is a bad misconception. Why would you want to search the PV one way, and then search the other moves another way? Isn't what is good for the PV good for the rest? Don't you want the rest to have an opportunity to fail high and replace the original PV? This is one of those ideas that simply makes absolutely no sense in any type of alpha/beta minimax tree search application. You can treat moves differently if you wish. But if it gets done differently based on PV vs non-PV that too is flawed.

I doubt we disagree, I just wasn't directly addressing only the PV case, which you were apparently responding to.

I just wondered if you also apply forward pruning at PV nodes (null moves, razoring etc.) if you don't want to treat them differently?

[jdart]

I see no reason to treat PV vs non-PV nodes any differently. That is a bad misconception. Why would you want to search the PV one way, and then search the other moves another way?

I suspect avoiding transposition table cutoffs in the PV became popular due to its use in Fruit. It doesn't make a lot of sense to me either.

But Fruit also uses other PV-dependent techniques for example, different extensions in full nodes vs. zero-width nodes. I think the general idea is that you are going to cut most of those nodes anyway, so you cut the tree size a bit. If the search throws up anything really interesting that node will get re-searched with the full extensions. A bit like LMR but playing with the extensions. This is now done in quite a few strong programs.

[michiguel]

I see no reason to treat PV vs non-PV nodes any differently. That is a bad misconception. Why would you want to search the PV one way, and then search the other moves another way?

I suspect avoiding transposition table cutoffs in the PV became popular due to its use in Fruit. It doesn't make a lot of sense to me either.

I think that for an engine that is used for analysis is the way around: It does not make any sense to gain almost nothing in terms of search speed and lose the PV, which is a valuable piece of information. People are crazy about gaining 1 elo point and forget how a true chess player may use the engine.

Miguel

But Fruit also uses other PV-dependent techniques for example, different extensions in full nodes vs. zero-width nodes. I think the general idea is that you are going to cut most of those nodes anyway, so you cut the tree size a bit. If the search throws up anything really interesting that node will get re-searched with the full extensions. A bit like LMR but playing with the extensions. This is now done in quite a few strong programs.

[yoshiharu]

I think that for an engine that is used for analysis is the way around: It does not make any sense to gain almost nothing in terms of search speed and lose the PV, which is a valuable piece of information.

I would also add that if your engine is still in a state of flow, having a complete PV being reported can help you trap some bug or weakness of the engine itself.
It lately helped me while bug-hunting.
And alas!, if I need help

Cheers, Mauro

[bob]

I was actually just responding to the "general case". I see no reason to treat PV vs non-PV nodes any differently. That is a bad misconception. Why would you want to search the PV one way, and then search the other moves another way? Isn't what is good for the PV good for the rest? Don't you want the rest to have an opportunity to fail high and replace the original PV? This is one of those ideas that simply makes absolutely no sense in any type of alpha/beta minimax tree search application. You can treat moves differently if you wish. But if it gets done differently based on PV vs non-PV that too is flawed.

I doubt we disagree, I just wasn't directly addressing only the PV case, which you were apparently responding to.

I just wondered if you also apply forward pruning at PV nodes (null moves, razoring etc.) if you don't want to treat them differently?

I apply _everything_ equally, yes. Forward pruning. Null-move. Reductions. All are done everywhere, with no exceptions that are based on CUT/ALL/PV node types.

[Eelco de Groot]

I see no reason to treat PV vs non-PV nodes any differently. That is a bad misconception. Why would you want to search the PV one way, and then search the other moves another way?

I suspect avoiding transposition table cutoffs in the PV became popular due to its use in Fruit. It doesn't make a lot of sense to me either.

I think that for an engine that is used for analysis is the way around: It does not make any sense to gain almost nothing in terms of search speed and lose the PV, which is a valuable piece of information. People are crazy about gaining 1 elo point and forget how a true chess player may use the engine.

Miguel

But Fruit also uses other PV-dependent techniques for example, different extensions in full nodes vs. zero-width nodes. I think the general idea is that you are going to cut most of those nodes anyway, so you cut the tree size a bit. If the search throws up anything really interesting that node will get re-searched with the full extensions. A bit like LMR but playing with the extensions. This is now done in quite a few strong programs.

I think those are important reasons, for me at least and Miguel's remarks are not technical, so not much dispute about that.

Another simple reason why PV- nodes and Non PV nodes are different; not that this is required per sé, but usually PV nodes will be searched by Aspiration window search, and Non PV nodes will be searched by a Null window search.

● The characteristics of these searches being -very?- different, this is more important than the other mentioned changes. It is not so much about treating the nodes differently, it is about the different types of search.

● Aspiration window search under circumstances sometimes can be very costly compared to Null window search. But this is usually offset by much smaller number of PV nodes. You can afford other costly measures better in PV nodes.

● In my view all the other differences are just ways of making these two types of searches work together better.

That does not mean any of the different ways of handling PV nodes and Non PV nodes -in Stockfish or other programs- are all necessary, or even any of them, but it just makes sense that PV nodes will be more valuable, otherwise they would not be PV nodes. There is nothing mysterious about that, I'm not claiming having new insights here but it is also no 'misconception'. In my humble opinion

Eelco