Mate solving test SF19, SF18 and SF11.

[Jouni]

600 longest positions from matetrack (mate in 16 to mate in 126) https://github.com/vondele/matetrack . Found mates in 10 second with 6 cores:

SF11 288
SF18 268
SF19 243

For comparison Huntsman 443.

[Jouni]

Stockfish 19 surprises by solving some classic positions very fast.

[d]3qk3/8/8/8/8/8/PPPPPPPP/RNBQKBNR w KQ - 0 1

Analysis by Stockfish 19:
...
1.e4 Qd6 2.d4 Qg6 3.Qf3 Qf7 4.Bb5+ Ke7 5.Bg5+ Kf8 6.Bh6+ Ke7 7.Qg3 Kd8 8.Qd6+ Kc8 9.Bf4 Qa7 10.Ba6+ Qxa6 11.Qc7#
+- (#11) Depth: 51/24 00:00:15 71308kN

[mysammoore]

As long as there are consistent checks, I would always imagine an HCE engine to out-calc mate lines vs. an NNUE unless the HCE was very slow. Mate lines are far more factual than eval-driven. Honestly the stronger the NNUE the weaker I'd expect it to perform under this test scenario.

[Eelco de Groot]

It is not really sure Stockfish 11 consistently outperforms Stockfish 19 even if you have a large sample like this. (It is still all a bit nondeterministic with 6 threads) The HCE helps of course if it is faster. Stockfish can speedup mate searches (just for testing?) and if I read the commit right, in matetrack.epd testing now does so by making search changes. If you know you are looking for mate you can for instance use a pretty narrow and high search window so everything else falls outside of it. I am not really sure why they are doing it if it is just to test Master. But if you put difficult positions in the epd they use, Stockfish will search some faster now but it is more for very short DTMs if I read that. Some people would like a version where they could set that variable themselves in UCI options perhaps

[Jouni]

Result is no surprise, because SF11 speed is triple to SF19! 10 MNPS vs 3,5 MNPS. Actually Stockfish 19 is best mate finder among top programs.

[Uri Blass]

Result is no surprise, because SF11 speed is triple to SF19! 10 MNPS vs 3,5 MNPS. Actually Stockfish 19 is best mate finder among top programs.

The question is if the difference is explained by difference in nodes per second and if SF19 get better results than SF11 if you increase the time of SF19 from 10 seconds per position to 30 seconds per position .

[Jouni]

I gave SF19 30 seconds, but it was still worse than SF11 with 10 s.

[Dann Corbit]

I think it has very little to do with NPS.

In general, each version of SF has a smaller branching factor than the previous versions. That means that they prune more. Stupid looking moves are pruned more vigorously. The thing that makes chess puzzles good puzzles is that the winning move is often stupid looking and/or involves sacrifices. The kind of move that gets pruned more aggressively.

I think the newer Stockfish versions are much better at quiet moves and worse at tactical puzzle moves. Since most moves in a chess game are quiet moves, that is why it wins.

[Jouni]

"I think the newer Stockfish versions are much better at quiet moves and worse at tactical puzzle moves." No. SF19 is best Stockfish tactically.

[Paloma]

Hmm, SF19 is 4th (last) in your ranking in your first post above.

Which Matetrack should I choose?
(Improve a mate in 21 to a mate in 20) Which FEN is this? Red, blue, or green?

Could you therefore please post your list of the 600 FENs?