Problems with pruning non-PV nodes

[OliverBr]

The following position 8/3q3k/1p1r3p/2rp1Rn1/p2NpQP1/2P3P1/PP6/3R3K w - - 2 38

[d]8/3q3k/1p1r3p/2rp1Rn1/p2NpQP1/2P3P1/PP6/3R3K w - - 2 38

has a strong singular winning move 38. Kg2 and this line:

Code: Select all

Kg2 Rg6 Rh1 Qe7 Re5 Qf6 Qxf6 Rxf6 Rxg5

Engines do have problems finding this move.
But, when you give the move to an engine, it finds immediately the winning line and correct score.

The reason for this is the pruning and search depth reductions of nodes that are non-PV.
Those things let the engines play stronger in general, but makes it makes them weaker to detect tactical surprises.

What is the general theory to this solve this issue?

[mar]

I'd like to know that too

I guess a better static evaluation might help for sure
better move ordering as well (especially for quiets because trying better moves first makes them less likely to be over-reduced/pruned in LMR/LMP)

non-pv nodes (zero-width scout) is how we find better moves
we tend to prune/reduce a lot these days and everything adds up (nullmove in scout nodes etc)
plus the depth we store in TT doesn't actually truly correspond to what we actually searched, because the subtrees are higly asymmetric,
extend here, reduce there, prune there => what I mean by that is that depth n doesn't mean we've exhaustively searched n plies at a given node

a general observation I made is that the better current (pv) move is, the harder it is to find a better move (assuming there is one, of course)
sometimes luck plays a role too (this is especially true in SMP searches)

I've added Ed's IIR recently and got 15 elo is hyperbullet (which is very nice - thanks, Ed), but it severely hurts cheng's ability to solve some tactical positions.
I tried a ton of things to remedy this but all of them lose elo. it drives me crazy and I've already wasted a ton of processing power on this.

I think that the vast majority of positions engines search in engine-engine matches is non-tactical in the end; and we get elo from ideas that work most of the cases. (this is also why a new endgame knowledge typically adds little elo, because you don't get many positions where you can apply it)

EDIT: one small story about delta pruning in qs: I've removed it in 4.41 because it didn't play well with endgame knowledge, I've since added it back
but now I add extra "positional margin": I have current pst+material for free (update incrementally in makemove), so I add abs(eval - pst_mat_only)
as and extra delta pruning margin - this should cover things like ks or endgame knowledge. I wonder if I could use something similar
in futility pruning/razoring without losing elo (doesn't look like it so far

my two cents

[OliverBr]

Thank you, @mar for your reply. You have got my issue on the point.

better move ordering as well (especially for quiets because trying better moves first makes them less likely to be over-reduced/pruned in LMR/LMP)

Absolutely correct!

a general observation I made is that the better current (pv) move is, the harder it is to find a better move (assuming there is one, of course)
sometimes luck plays a role too (this is especially true in SMP searches)

Yes, this is my observation, too.
In my example position there are other moves that also win the game. In those situations the very best move is hardly never found, but doesn't bother in the overall ELO.

I've added Ed's IIR recently and got 15 elo is hyperbullet (which is very nice - thanks, Ed), but it severely hurts cheng's ability to solve some tactical positions.

Ed's IIR severely weakens non-hash (aka surprising) nodes, but it's make the engine stronger generally. It almost looks like a paradox.

my two cents

Very much appreciated!