Stockfish 15.1 is ready

[pohl4711]

Ratinglist- and regression-testruns of Stockfish 15.1 finished.


https://www.sp-cc.de

Also take a look at the EAS-Ratinglist, the world's first engine-ratinglist not measuring strength of engines but engines's style of play:
https://www.sp-cc.de/eas-ratinglist.htm

(Perhaps you have to clear your browsercache (press <STRG>+<SHIFT>+<DEL>) or reload the website))

[Magnum]

After 568 games on CCRL 4015 one thread, it is +28 elo (bayeselo) and +37 Ordo elo. I'd say those are good gains for what was already an incredible engine.

https://www.chessengeria.com/post/stoc ... 1-released

Stockfish 15.1 compiled for Macs is noticeably stronger than its predecessor.

Great news. This may add weight to Magnum's view that previous compiles weren't that good.

One way the SF Dev team could show they care about Apple Silicon is by releasing compiles as standard for it, as they already do for Windows and Linux

See the MacRumors forum.
They have probably 100 sites about the Stockfish engine on Apple M1 macOS.
You should really read all of them.
Result: Stockfish can run on the new M1 chip. But Stockfish on Intel or AMD was improved over 14 years. The Stockfish team can't put over 14 years on Intel/AMD in 1 day to the new M1 architecture. Another problem is that most of them doesn't have an M1, M1 Pro, M1 MAX, M1 ULTRA, for many reasons like (better not buy the first generation and start with the second generation M2 MAX or ULTRA or EXTREME) but the number of Stockfish developers with M1.. is slowly increasing.

SF dev team could show that they care about Windows 11 ARM and releasing compiles as standard for it.

[Modern Times]

After 568 games on CCRL 4015 one thread, it is +28 elo (bayeselo) and +37 Ordo elo. I'd say those are good gains for what was already an incredible engine.

We have a regression of approx. 20 ELO after 800 games
(only!) with shorter TC
https://cegt.forumieren.com/t1830-testi ... ish-15-1nn

I've just run over 1,000 games at blitz 2+1 and got +25 Elo (bayeselo, Ordo would probably be more)
Pohl SuperGM4moves, Intel i9 10900

[Modern Times]

I've spent quite a bit of time on the phone with Apple support who were fairly obstructive. It seems clear they want every piece of software to go via them.

Are you surprised by that ? Typical Apple behaviour from what I've heard. Lock you in, take away your freedom to do things how you want to do them, and make you do them their way or not at all. No wonder the SF team do not seem keen to support Apple, it goes against everything they believe in with open source philosophy.

I suppose not but I am saddened because I do think they've achieved something with their hardware which could benefit the chess world (for once). The decision to not release SF Silicon Apple builds seems to me a little like cutting one's nose to spite one's face.
But I can understand the anti-Apple sentiment.

Agreed on the hardware front, M1 and M2 are very impressive CPUs for the balance of power and performance. Great job by Apple but no surprise really, they have always made good hardware.

[Harry T]

I would like more info on the new eval scheme as well. Through some testing, it looks like the scale goes from -88 (Bk is lost) to +88 (Wt wins).

Does the +1 score also mean there is a 48% (perhaps) chance of draw, and 2% chance of loss? Why invent a new scale for this; why not just use % chance of winning, or "expected score" as AlphaZero uses?

Any opinion about this new evaluation calibration? (or however this should be called):

"An evaluation of +1 is now no longer tied to the value of one pawn, but to the likelihood of winning the game. With a +1 evaluation, Stockfish has now a 50% chance of winning the game against an equally strong opponent."

[Ajedrecista]

Hello Harry:

[...]

Does the +1 score also mean there is a 48% (perhaps) chance of draw, and 2% chance of loss? Why invent a new scale for this; why not just use % chance of winning, or "expected score" as AlphaZero uses?

[...]

Running SF 15.1 under Arena 2.0.1 GUI on Windows, there is a checkbox with the option UCI_ShowWDL that does not work for me when ticked, like other SF releases. In other words:

☑ UCI_ShowWDL

Is the same for me under Arena 2.0.1 GUI than:

☐ UCI_ShowWDL

Showing the normal evals in pawns. However, if you run SF 15.1 through the command prompt (double click/press Enter on the executable) and type the uci command, SF will show you the UCI options:

Code: Select all

Stockfish 15.1 by the Stockfish developers (see AUTHORS file)
uci
id name Stockfish 15.1
id author the Stockfish developers (see AUTHORS file)

option name Debug Log File type string default
option name Threads type spin default 1 min 1 max 1024
option name Hash type spin default 16 min 1 max 33554432
option name Clear Hash type button
option name Ponder type check default false
option name MultiPV type spin default 1 min 1 max 500
option name Skill Level type spin default 20 min 0 max 20
option name Move Overhead type spin default 10 min 0 max 5000
option name Slow Mover type spin default 100 min 10 max 1000
option name nodestime type spin default 0 min 0 max 10000
option name UCI_Chess960 type check default false
option name UCI_AnalyseMode type check default false
option name UCI_LimitStrength type check default false
option name UCI_Elo type spin default 1350 min 1350 max 2850
option name UCI_ShowWDL type check default false
option name SyzygyPath type string default <empty>
option name SyzygyProbeDepth type spin default 1 min 1 max 100
option name Syzygy50MoveRule type check default true
option name SyzygyProbeLimit type spin default 7 min 0 max 7
option name Use NNUE type check default true
option name EvalFile type string default nn-ad9b42354671.nnue
uciok

Then you can change UCI_ShowWDL command (false by default) with the following line:

Code: Select all

setoption name UCI_ShowWDL value true

And analysis of positions will show WDL estimates in per mille units (‰). For example, going up to depth 6 in the starting position:

Code: Select all

go depth 6
info string NNUE evaluation using nn-ad9b42354671.nnue enabled
info depth 1 seldepth 1 multipv 1 score cp 18 wdl 22 974 4 nodes 20 nps 1176 hashfull 0 tbhits 0 time 17 pv e2e4
info depth 2 seldepth 2 multipv 1 score cp 46 wdl 82 917 1 nodes 66 nps 3300 hashfull 0 tbhits 0 time 20 pv d2d4
info depth 3 seldepth 2 multipv 1 score cp 51 wdl 105 894 1 nodes 120 nps 5714 hashfull 0 tbhits 0 time 21 pv e2e4
info depth 4 seldepth 2 multipv 1 score cp 58 wdl 140 859 1 nodes 144 nps 6857 hashfull 0 tbhits 0 time 21 pv d2d4
info depth 5 seldepth 2 multipv 1 score cp 58 wdl 140 859 1 nodes 174 nps 8285 hashfull 0 tbhits 0 time 21 pv d2d4 a7a6
info depth 6 seldepth 7 multipv 1 score cp 34 wdl 48 950 2 nodes 1303 nps 54291 hashfull 0 tbhits 0 time 24 pv e2e4 c7c5 g1f3 b8c6 c2c3
bestmove e2e4 ponder c7c5

Where wdl 48 950 2 must be understood as W=48‰ (for the side to move), D=950‰ and L=2‰ (W = 4.8%, D=95%, L=0.2%). Going up to depth 30 with single core (default in command prompt, aiming for determinism) and default hash (16 MB) shows:

Code: Select all

go depth 30
[...]
info depth 30 seldepth 34 multipv 1 score cp 28 wdl 36 962 2 nodes 10228560 nps 457121 hashfull 1000 tbhits 0 time 22376 pv d2d4 d7d5 c2c4 c7c6 g1f3 g8f6 b1c3 d5c4 a2a4 c8f5 e2e3 b8d7 f1c4 e7e6 e1g1 f5g6 d1e2 f8b4 c4d3 g6d3 e2d3 d8c7 e3e4 e6e5 d4e5 d7e5 f3e5
bestmove d2d4 ponder d7d5

W=3.6% (for the side to move), D=96.2%, L=0.2%.

Analysing the famous Shirov's 47.- ..., Bh3 with single core (aiming for determinism), 512 MB of hash and without EGTB up to depth 34:

Code: Select all

setoption name Hash value 512
position fen 8/8/4kpp1/3p1b2/p6P/2B5/6P1/6K1 b - - 2 47
go depth 34
[...]
info depth 34 seldepth 53 multipv 1 score cp 660 wdl 1000 0 0 nodes 24990696 nps 787803 hashfull 165 tbhits 0 time 31722 pv f5h3 g1f2 e6f5 g2h3 f5e4 c3f6 d5d4 f6e7 e4d3 e7b4 d3c2 f2f3 d4d3 f3e3 a4a3 b4a3 d3d2 e3f4 d2d1q f4g5 d1h5 g5f6 h5f3 f6g6 f3a3 g6f6 a3e3 f6f5 e3h3 f5f6 h3h4 f6e5 h4h6 e5d4 c2b3 d4e5 h6h5 e5d4 h5h8 d4c5
bestmove f5h3 ponder g1f2

Totally winning for black (wdl 1000 0 0).

Regards from Spain.

Ajedrecista.

[wickedpotus]

Agreed on the hardware front, M1 and M2 are very impressive CPUs for the balance of power and performance. Great job by Apple but no surprise really, they have always made good hardware.

Apple Hardware is sadly also overrated and overpriced.. Nothing special apart from the hyped "influencer" marketing on social-media... Both GPU-bound engines and CPU-bound engines are much much faster on 12th gen Intel and even last gen AMDs on mobile, even with inexpensive RTX-gpus these days at the pricepoints given.

[wickedpotus]

See the MacRumors forum.
They have probably 100 sites about the Stockfish engine on Apple M1 macOS.
You should really read all of them.
Result: Stockfish can run on the new M1 chip. But Stockfish on Intel or AMD was improved over 14 years. The Stockfish team can't put over 14 years on Intel/AMD in 1 day to the new M1 architecture. Another problem is that most of them doesn't have an M1, M1 Pro, M1 MAX, M1 ULTRA, for many reasons like (better not buy the first generation and start with the second generation M2 MAX or ULTRA or EXTREME) but the number of Stockfish developers with M1.. is slowly increasing.

SF dev team could show that they care about Windows 11 ARM and releasing compiles as standard for it.

BS. SF is open-source and Apple's GPUs or CPUs with its limited single-threaded-optimized designfocus (no hyperthreading support etc) are simply not very competitive on higher-end mulithreaded max performance... There are most likely even more potential optimizations to do on new intel and AMDs richer instruction sets than there are for the more instruction-reduced ARM architectures. But regardless we will be talking about impromant of a few percen not compensating for M1 posting benches half of that of competitors laptops.

If you claim an M1/M2 somehow even potentially could beat an i7-1200H (it pretty constantly gets beaten on most multithreaded benchmarks even outside the realm of high-perfromance chess-engines) with some unlikely optimizations, then just prove it yourself, or stop making such claims..

Fairy-tail fanboy fantasyland and space-gray unicorns are for other forums than talkchess

[M ANSARI]

Agreed on the hardware front, M1 and M2 are very impressive CPUs for the balance of power and performance. Great job by Apple but no surprise really, they have always made good hardware.

Apple Hardware is sadly also overrated and overpriced.. Nothing special apart from the hyped "influencer" marketing on social-media... Both GPU-bound engines and CPU-bound engines are much much faster on 12th gen Intel and even last gen AMDs on mobile, even with inexpensive RTX-gpus these days at the pricepoints given.

I don't agree at all about the Apple hardware statement. I never really took notice of all the Apple hype, but my cousin happened to be the local agent for Apple in my country and I ended up getting one of the new Macbook Pro's 13 inch with the new M2 chip. OMG !!! I cannot explain how incredibly impressed I was with the video editing performance of my spearfishing GoPro footage. It was dramatically faster ... I mean by several orders of magnitude ... than my souped up 8 core PC with a 1500 watt power supply and a 2080Ti card. Even 4K videos would scream through it like it was nothing. Totally amazing ... and all that without the laptop even getting warm ... with battery life that lasted the full day. So I personally was absolutely impressed with the performance.

I will add however that I hardly ever use that laptop except for video editing. I just can't get around the learning curve required to learn the new OS as I have been using DOS then Windows since my first PC. It does get me frustrated when I want to do a simple task ... but I am sure that can be resolved once I put some time and effort into it. I also don't know how well the M2 chip performs for chess as I was not able to run the Ipman test on it ... but if video editing performance is comparable to chess engine performance ... that thing will fly!

[Graham Banks]

After 568 games on CCRL 4015 one thread, it is +28 elo (bayeselo) and +37 Ordo elo. I'd say those are good gains for what was already an incredible engine.

We have a regression of approx. 20 ELO after 800 games
(only!) with shorter TC
https://cegt.forumieren.com/t1830-testi ... ish-15-1nn

Funnily enough my 4CPU results for SF 15.1 show a slight regression (around 10 Elo) so far as well, but only 120 games played, so the error bars are still big. Many more games to go.