Natural TB

[syzygy]

But this alone does not fix the issues I'd like to target with my patch: mates, sacrifices, etc. In particular as long as the search does not find a mate PV, the "naked" TB scores and behavior will be exposed to user.

Which seems to be exactly what the user would want (although it would be nicer to see "TB win in 5 moves" instead of a very large score, but this is a GUI issue).

[syzygy]

That's why I search swallow: I already know a priori it will not fail low. This is a very important information. Reason why engines search very deep the PV is not to find a better PV, but to prove the current PV does not fail low. We already know it after a TB probing, so we implicitly use this to do a swallow search and save resources.

First, as far as I can tell this is not restricted to searching the PV (in fact you don't even probe in the PV, no doubt greatly confusing the search if the root position has a forced but deep TB win).

Second, what if the shallow search does fail low? Although you know it would not fail low if searched at infinite depth, the shallow search may very well fail low. And you're doing nothing to correct that.

Third, obviously you are not saving resources if you do a search where you could simply have returned the TB score immediately.

[syzygy]

Yes, I treat them differently, and not only when entering in TB, but always

The reason for this is to avoid odd sacrifices and TB scores to be exposed to user. If I don't overwrite search value in case of a winning position I assume the above two "features" of TB behavior will not be exposed.

Now what if the winning side's recapture after the queen sac is the move that enters the TB? The TB probe will now return a TB loss, confirming that the queen sac is winning.

So again, there is no good reason to treat the two cases differently.

Of course by not probing in the PV you'll avoid TB scores in PV lines. But that has nothing to do with treating TB wins differently from TB losses. (And of course not probing in the PV comes with its own set of undesirable effects.)

[mcostalba]

(And of course not probing in the PV comes with its own set of undesirable effects.)

I don't probe in PV for 2 reasons:

1. Align to what we do in TT, where we don't cut-off in PV after a TT probing (this is not mandatory, many engines do cut-off and also SF in the past did, but we move to current scheme because of truncated PV and because don't cut-off in PV did not give any ELO regression).

2. Avoid truncated lines in lose positions, when we return immediately with the TB score. I experimented that without disabling probing in PV I was not able to find mate lines in simple positions.

Maybe there are other ways to avoid truncated mate lines, but this is simple and effective.

I am curious about the "undesirable effects" you are referring. Can you please better explain which are the undesirable effects of disabling TB probing in PV?

[mcostalba]

Something else I do in Matefinder is to not stop probing even after we reach TBs in Root. Since you no longer return early for non-draws, you still search, but you get a nice speed up (in depth) due to all the draw cutoffs.

This seems a nice idea.

Indeed with natural TB scheme I am thinking of removing the root moves filtering because now it is largely (not 100%) just a special case of current setup.

Current code, if root position is in TB, filters out bad root moves but then disables probing in the sub-tree. Disabling probing in the sub-tree after a winning TB probe it is already included by default in natural TB, so this special treatment of root position is now largely redundant and I am greatly tempted to drop it.

What stopped my until now is that we lose DTZ probing and we rely just on WDL, although I bet for real games this should not matter a lot because theoretically, because we allow the search to continue after a winning position, we are able in infinite time, to find anything that DTZ knows in advance. So there is not a theoretical loss of information in dropping DTZ under natural TB scheme (not in current scheme where WDL probing could return win when it is not due to move 50 rule).

[hgm]

Sounds that you should never be able to find a TB-win that way, only mates and heuristic evaluations. If the mate is not within the horizon, the PV move would always return a heuristic score. Alternative moves in PV nodes will be able to use probing in their sub-tree, see the TB-win, and thus fail high. Then they will be re-searched as PV move, no longer probe, and again return a heuristic score.

Eventually every move will be searched as PV move, (i.e. without any EGT probing), like in plain alpha-beta, but with a preceding null-window search that will always fail high.

[mcostalba]

see the TB-win, and thus fail high.

See the tb-win and DO NOT fail high (but keep searching at reduced depth and return normal search score).

[syzygy]

I don't probe in PV for 2 reasons:

1. Align to what we do in TT, where we don't cut-off in PV after a TT probing (this is not mandatory, many engines do cut-off and also SF in the past did, but we move to current scheme because of truncated PV and because don't cut-off in PV did not give any ELO regression).

Not taking TT cutoffs in the PV makes sense because most values stored in the TT come from non-PV searches, which are done with more reductions and less extensions and therefore are less reliable than the PV search they would replace if the cutoff were taken.

Not taking TT cutoffs also has an obvious cost, but apparently this cost does not outweigh the above-mentioned benefit.

2. Avoid truncated lines in lose positions, when we return immediately with the TB score. I experimented that without disabling probing in PV I was not able to find mate lines in simple positions.

That your particular experiment did not work out does not say much.

Maybe there are other ways to avoid truncated mate lines, but this is simple and effective.

Removing all TB probing code is even more simple and effective in this regard...

I am curious about the "undesirable effects" you are referring. Can you please better explain which are the undesirable effects of disabling TB probing in PV?

Obviously disabling TB probing introduces search instability and it prevents SF from finding the winning line to a TB win.

[mcostalba]

That your particular experiment did not work out does not say much.

Unfortunately your answer say even less.

Removing all TB probing code is even more simple and effective in this regard...

Ditto.

Obviously disabling TB probing introduces search instability and it prevents SF from finding the winning line to a TB win.

It is too much to ask for an example position? The only obvious thing is that for me and you the word "obvious" has a different meaning.

Here it is common for people (not you in particular) to invoke "search instability" as a kind of amulet when they have no more arguments, it is a kind of last resort answer, is like the shooting ink of the octopus

[hgm]

see the TB-win, and thus fail high.

See the tb-win and DO NOT fail high (but keep searching at reduced depth and return normal search score).

So you never use the EGT score anywhere, the only way you ever use a probe is to prune non-wins, and reduce wins?

Well, if the root is a win then every searched line will be reduced, and reducing everything doesn't alter the shape of the tree, but just causes the engine to lie more about the depth it reaches. So it would become equivalent to a WDL-guided search.

If the root is a win not yet in any EGT, all lines going in the EGT will be reduced, but the lines that avoid conversion will not. If the conversion itself is no obvious progress (i.e. a trade, rather than gaining material), this will make it less likely that it will go for a conversion, as when you are winning, the deeper you search, the better the score usually gets.