1.g4 opening is losing?

[zullil]

No line appears in your engine output. All I see for the Mate-in-26 is a four-move PV. How do you know what line to force if you don't know the line that is the engine has called Mate-in-26?

I'm using two engines. The other one knows a line that mates at all times (though not the fastest mate, just the easiest mate to find), so I feed the mating moves to this one. The difference is that this one has learning, the mating one doesn't, so eventually this one would learn a mate score and be able to show it from the root (I'm making it show mate scores closer and closer to the root) while the mating one would always have a mate outside of its horizon from the root, because of the depths I can reach.

So you have no idea what line "Harem Girl" is actually considering when it shows Mate-in-26 and a four-move PV. Clearly that engine found a line that it has mistakenly evaluated as Mate-in-26, and is in the process of failing high as it begins to correct its error. Somewhere in the tree, the engine has come upon a better defense for Black than what it saw originally. It's not saying that the position is suddenly drawn; it's simply reassessing the true score of the position. Eventually another mate score should appear.

[jp]

Clearly that engine found a line that it has mistakenly evaluated as Mate-in-26, and is in the process of failing high as it begins to correct its error. Somewhere in the tree, the engine has come upon a better defense for Black than what it saw originally. It's not saying that the position is suddenly drawn; it's simply reassessing the true score of the position. Eventually another mate score should appear.

So in general when this sort of thing happens, do you favor "engine mistake" over "engine forgetting"?

[zullil]

So when an engine shows mate in 30, and this blows out to mate in 50, etc., you believe it's always engine "forgetting" rather than engine mistake?

I've never seen a mate in 30 reported when it was impossible to mate this fast, have you?

Saw this happen just yesterday with asmfish as it searched Zenmastur's position. Engine showed mate-in-35, which then rose as high as mate-in-38 during additional iterations of search, only to return eventually to mate-in-35 (but with a different line than it had earlier).

[zullil]

Clearly that engine found a line that it has mistakenly evaluated as Mate-in-26, and is in the process of failing high as it begins to correct its error. Somewhere in the tree, the engine has come upon a better defense for Black than what it saw originally. It's not saying that the position is suddenly drawn; it's simply reassessing the true score of the position. Eventually another mate score should appear.

So in general when this sort of thing happens, do you favor "engine mistake" over "engine forgetting"?

I think that the ++ notation attached to the scores in his output indicate that the engine is in the process of failing high, i.e., it's realizing that it missed a refutation and is in the process of correcting a mistake.

[jp]

I think that the ++ notation attached to the scores in his output indicate that the engine is in the process of failing high, i.e., it's realizing that it missed a refutation and is in the process of correcting a mistake.

Yes, I think that's what the notation means, but normally when I've seen this behavior (typically from Sesse), it does not give any hints about what is happening. It just flashes a mate score and PV, and then flashes another mate score and another PV, all very quickly.


Here are Uri's earlier thoughts, in case anyone interested missed them.

1)No

It does not mean that the mate in 30 was just wrong(I do not know and maybe stockfish has memory problem and does not memorize the mating line and prune it at bigger depth).
Of course hash collision is also possible but I never saw a single case of reproducable analysis of stockfish that show a wrong mate because of hash collision.

In order to prove the mate in 30 was just wrong you need to prove that there is no mate in 30.

I would like to see a position when stockfish claim mate in 30 when there is no mate in 30.
Maybe somebody can try all the mate positions from tablebases to see if there is a single position when stockfish without tablebases claim mate that is shorter than the shortest mate(when we assume for the discussion that tablebases are right about the shorter mate).

[Ovyron]

I don't know. I hope not! That's the open question.

Sometimes SF makes the mate announcement and then calculates longer and makes a longer mate announcement or retracts altogether. It must raise some doubt. We need to capture examples to look at.

I also wonder when it gives a mate announcement with a mate line that is apparently "wrong" (White and Black trade suboptimal moves). See here: http://talkchess.com/forum3/viewtopic.p ... 94#p829094.

It's what we want when an engine updates its mate announcement to a quicker mate, but if that means it found better moves for the mating side, why would it be impossible for it also to find better moves for the defending side (leading to a slower mate) -- unless it becomes near-exhaustive in analyzing defending moves once it thinks there's a mate?

Yeah, the engine doesn't report mate unless it has exhausted the opponent defenses, and doesn't report that a mate is at least in N unless it has exhausted them and can at least guarantee a mate in N would happen. As the exhausted lines are overwritten in the TT it forgets it has seen them, so the mate may go to a higher count, or if all of them are overwritten, it might disappear. But it was there at the time it was reported, and there's no doubt a mate exist and it's at least that short (though it can be shorter.)

If you want to investigate, I recommend giving few RAM to the engine, like, 8MB RAM or something, so the mates it sees can't be stored and this behavior is maximized.

[jp]

Yeah, the engine doesn't report mate unless it has exhausted the opponent defenses, and doesn't report that a mate is at least in N unless it has exhausted them and can at least guarantee a mate in N would happen.

Is that really how "all" engines are programmed? It can't be completely exhaustive, though, can it? (If it is, we just need it to output its results and we have the desired mate tree.) I'd like to see the source code in SF where such a change to pruning occurs.

And if it spits out a mate line with the mate announcement that has cancelling suboptimal moves by White and Black that still doesn't give me confidence.

As the exhausted lines are overwritten in the TT it forgets it has seen them, so the mate may go to a higher count, or if all of them are overwritten, it might disappear.

If you want to investigate, I recommend giving few RAM to the engine, like, 8MB RAM or something, so the mates it sees can't be stored and this behavior is maximized.

But both "forgetting" and "mistake" could be possible, so just generating instances of "forgetting" doesn't rule out other instances of "mistake".

[Ovyron]

Somewhere in the tree, the engine has come upon a better defense for Black than what it saw originally.

Nope, it has forgotten what moves white was using for the mate, so now it can no longer see the mate, and this produces a fail high, and now the best white moves it can find are a +10.00 eval. The other engine doesn't forget the white moves (but it'd forget them if I go back in the tree and the TT fills with new positions), so feeding them back to "Harem Girl" makes her catch up with them and report the right mate score again. But no better black defense exists (because a mate in 25 is the lowerbound, white may mate faster, but black can't delay the mate more moves, if this wasn't so the engine wouldn't have shown it in the first place.)

[Ovyron]

Yeah, the engine doesn't report mate unless it has exhausted the opponent defenses, and doesn't report that a mate is at least in N unless it has exhausted them and can at least guarantee a mate in N would happen.

Is that really how "all" engines are programmed?

It's how they're supposed to be programmed, if an engine shows a Mate in N when it doesn't exist or N is higher in reality, it's a bugged engine. These engines here aren't bugged, so a Mate in 25 can't be improved by black if it's reported.

It can't be completely exhaustive, though, can it?

Yes, it can, it has searched a half billion nodes and they include all possible black replies. It hasn't exhausted white attacks, so the mate could be done faster (that's what this challenge is about, me getting within 5 moves from reality.)

But both "forgetting" and "mistake" could be possible

No mistake can be possible unless there's a bug. There's nothing special about mates, so if you ever find an example of an engine reporting mate for a drawn position or a mate in N for a position that is a mate in N+x, you've found a bug, it's impossible that this happens in the search.

[jp]

But both "forgetting" and "mistake" could be possible

No mistake can be possible unless there's a bug.

That kind of mistake would not be a bug. It's just pruning. So unless seeing mate triggers code that makes it stop pruning for the defender I don't see why it cannot happen. Are you really claiming the engine is generating an entire mate tree with no defence move pruning, whenever it announces mate?