Stockfish and Tactics

[Werewolf]

A little moan:
I know it's a marvel that SF is able to search so deeply, prune so thoroughly, and still be tactically strong.

However, I am noticing tactical blind spots more frequently and wondered if others did too?

I tested the latest Dev (16th Nov 2023) build on a 64 core threadripper running a (sorry private) tactical testsuite.
On the threadripper it was searching at about 40 MN/s.

Giving 2 minutes per position, Stockfish scored 89/100.
Increasing the time to 10 minutes per position it only scored 91/100.

On several positions, the old Rybka Cluster from 10 years ago (140 cores) was significantly faster. In one position the Hydra FPGA supercomputer from 2005 was much faster.

As a reference, I tested SF Cluster via Chessify on 160 cores and then 768 cores and they scored 98/100 and 99/100 respectively (the bigger cluster achieved a significantly faster solve time). They were only allowed 2 mins per position. It made me wonder if SF Cluster has a broader search because the smaller cluster searching for 2 minutes should search fewer nodes than the 64 core threadripper at 10 minutes.

I'm now going to go back and test Rybka and Houdini on regular hardware. But I am coming to this conclusion: Stockfish's incredible depth comes at a cost...

[Ciekce]

sf's development is focused on elo gain, testsuites are completely irrelevant for modern engine dev

if a change makes stockfish """tactically weaker""" but better at actually winning games, that change still gets merged

also depth is a meaningless metric that cannot be compared between engines

[Werewolf]

Houdini 6 just scored 88/100.

As to the above post: yes granted.

Odd that the SF Cluster is massively outperforming regular SF though even allowing for the speed difference...

[smatovic]

...
Odd that the SF Cluster is massively outperforming regular SF though even allowing for the speed difference...

...as you mentioned, the parallel cluster search must widen the search in regard of pruning and selectivity (probably no LazySMP across cluster nodes?), how many nodes with cores did you run? You can verify this by giving non-cluster the NPS-equiavalent of time per position.

***edit***
It's called LazyMPI, Peter Österlund's "Lazy Cluster" algorithm:

Lazy SMP and "lazy cluster" experiments
https://www.talkchess.com/forum3/viewtopic.php?t=64824

https://github.com/official-stockfish/S ... /pull/1571
https://github.com/official-stockfish/S ... /pull/1931

--
Srdja

[Jouni]

Can You run SF Cluster code in threadripper!? Please post your results to SF discord where the real experts are.

[Werewolf]

Can You run SF Cluster code in threadripper!? Please post your results to SF discord where the real experts are.

How though? I looked at the cluster code before and it was too much. I don’t want to have to self compile

[Jouni]

Please test Crystal 7. It's only -16 elo from SF16.

[Werewolf]

Please test Crystal 7. It's only -16 elo from SF16.

On exactly the same hardware etc, Crystal 7 scored:

92/100 with an average solve time of 3.13 seconds

Overall I'd say it is a bit stronger than SF tactically and a bit weaker in terms of play (Elo).

[Werewolf]

...
Odd that the SF Cluster is massively outperforming regular SF though even allowing for the speed difference...

...as you mentioned, the parallel cluster search must widen the search in regard of pruning and selectivity (probably no LazySMP across cluster nodes?), how many nodes with cores did you run? You can verify this by giving non-cluster the NPS-equiavalent of time per position.

***edit***
It's called LazyMPI, Peter Österlund's "Lazy Cluster" algorithm:

Lazy SMP and "lazy cluster" experiments
https://www.talkchess.com/forum3/viewtopic.php?t=64824

https://github.com/official-stockfish/S ... /pull/1571
https://github.com/official-stockfish/S ... /pull/1931

--
Srdja

Isn't that what I explained in the first post?

Regular SF on threadripper at 40 MN/s at 10 mins/move allows for:
600 (seconds) x 40MN = 24 BN.

The Biggest Cluster Stockfish at 1.5BN/s had 2 mins per position:
120 (seconds) x 1.5BN = 180 BN.

The Smallest Cluster (160 cores) Stockfish at 300 MN/s had 2 mins per position:
120 (seconds) x 0.3BN = 36 BN.

The smallest cluster is only searching +50% more than my threadripper overall, but vastly outperforms it in tactics. I can't tell if the depths are comparable across these different versions. If so, the SF cluster is less impressive.

[smatovic]

...
Isn't that what I explained in the first post?
....

Ahh, then you already have your answer.

I did not look into SF Cluster code regarding LazyMPI, according to Peter Österlund's post on "Lazy Cluster" there are two different implementations possible.

--
Srdja