Stockfish 15.1 is ready

[Jouni]

SF 15.1 is worse than 15 in test suites. The reason is just one patch, that give +0,3 Elo or something like that .

[Eduard]

Complete agreement! That's why I prefer to build my own engine now. My modified Brainlearn 20.1 engine Vulkan 021222 is the best in my test, and clearly better than Stockfish 15.1.

By the way: Allegedly, 1.2 billion test games were played with Stockfish dev from 15.0 to SF 15.1 at Fishtest. Progress is almost zero. It can't be due to missing hardware.

[Modern Times]

By the way: Allegedly, 1.2 billion test games were played with Stockfish dev from 15.0 to SF 15.1 at Fishtest. Progress is almost zero. It can't be due to missing hardware.

It isn't almost zero. It is dependent on test conditions - book, hardware, time control etc. Book probably the most important one. Is a ratings list with one or two thousand games and big error margins more reliable than the 1.2 billion test games on fishtest ? My view is that the gains are there, 20-30 Elo, even if the test conditions aren't conducive to them being surfaced.

[Hai]

By the way: Allegedly, 1.2 billion test games were played with Stockfish dev from 15.0 to SF 15.1 at Fishtest. Progress is almost zero. It can't be due to missing hardware.

If you donate more hardware then the Stockfish developers can finish 1000x more tests in the same time and you will get the elo improvement.

[Eduard]

I always try to use the latest patches. But something is constantly being corrected, and the new corrected again and the old implemented again. Here's an example, it's not the only one of its kind. It doesn't surprise me that there is no progress.

New 081222:


Author: FauziAkram
Date: Thu Dec 8 20:41:45 2022 +0100
Timestamp: 1670528505

doEvenDeeperSearch + tuning

Passed STC:
LLR: 2.93 (-2.94,2.94) <0.00,2.00>
Total: 330048 W: 87672 L: 86942 D: 155434 Elo +0.77
Ptnml(0-2): 1012, 36739, 88912, 37229, 1132
https://tests.stockfishchess.org/tests/ ... 24c4c621d2

Passed LTC:
LLR: 2.95 (-2.94,2.94) <0.50,2.50>
Total: 216696 W: 57891 L: 57240 D: 101565 Elo +1.04
Ptnml(0-2): 72, 21221, 65152, 21790, 113
https://tests.stockfishchess.org/tests/ ... f096c68fe2


New (removed today) 121222:

Author: Joost VandeVondele
Date: Mon Dec 12 08:14:26 2022 +0100
Timestamp: 1670829266

Revert "doEvenDeeperSearch + tuning"

[mehmet123]

The time controls used in Fishtest should be changed to solve the power increase problem in Stockfish. 30 + 0.3 seconds and 180 + 2 seconds can be used instead of 10 + 0.1 seconds and 60 + 0.6 sec. Patch testing time will be longer, but we may see better quality patches.

[Paloma]

.
+1

[peter]

SF 15.1 is worse than 15 in test suites. The reason is just one patch, that give +0,3 Elo or something like that .

Not really worse but not really better neither. As for STC the smaller NNUE of SF15 could be of advantage yet still too.

Nr. 14+16

Code: Select all

    Program                                    Elo   +/-  Matches  Score   Av.Op.   S.Pos.   MST1    MST2   RIndex

  1 HypnoSFmpv210922-Set1-ImbInv             : 3568    3   9341    60.4 %   3494   206/256    1.8s    2.4s   0.73
  2 ShashChess26.1-GoldDigger-MV4            : 3562    3   9095    59.7 %   3494   195/256    1.7s    2.5s   0.72
  3 ShashChess26.2-MV4                       : 3562    3   9098    59.7 %   3494   198/256    1.8s    2.5s   0.74
  4 Crystal5KWK-MV4                          : 3561    3   9110    59.5 %   3494   196/256    1.7s    2.5s   0.71
  5 CorChess3dev-20221210-ec6b52d4-MV4       : 3561    3   9076    59.5 %   3494   198/256    1.8s    2.6s   0.72
  6 BlueMarlin15.3-MV4                       : 3560    3   8928    59.4 %   3494   192/256    1.7s    2.5s   0.73
  7 ShashChess25.4-MV4                       : 3560    3   9000    59.4 %   3494   193/256    1.7s    2.5s   0.73
  8 BlueMarlin15.4-avx2-MV4                  : 3559    3   8814    59.2 %   3494   189/256    1.6s    2.5s   0.74
  9 EMAN8.6064-bitBMI2-MV4                   : 3554    3   8749    58.5 %   3494   192/256    1.8s    2.6s   0.67
 10 CorChess3300522-MV4                      : 3553    3   8798    58.3 %   3494   189/256    1.8s    2.6s   0.68
 11 EMAN8.40-Tact.7-Expl.12-MV4              : 3550    3   8628    57.9 %   3495   183/256    1.7s    2.7s   0.70
 12 Stockfish110922-MV4                      : 3547    3   8508    57.4 %   3495   178/256    1.7s    2.7s   0.69
 13 Stockfish231022-MV4                      : 3545    3   8501    57.2 %   3495   180/256    1.8s    2.8s   0.68
 14 Stockfish15.1-MV4                        : 3545    3   8492    57.2 %   3495   178/256    1.8s    2.7s   0.68
 15 EMAN8.30-MV4                             : 3543    3   8389    56.9 %   3495   178/256    1.8s    2.8s   0.66
 16 Stockfish15-MV4                          : 3543    3   8462    56.8 %   3495   180/256    1.9s    2.8s   0.67
 17 Dragon3.1byKomodoChess64-bit-Set5-MV4    : 3523    3   8013    53.7 %   3498   154/256    1.7s    3.0s   0.61
 18 Dragon3.1byKomodoChess64-bit-MV4         : 3514    3   7804    52.2 %   3499   145/256    1.6s    3.1s   0.57
 19 Berserk10-MV4                            : 3488    4   7308    48.1 %   3502   124/256    1.8s    3.4s   0.49
 20 Ethereal13.75(NNUE)-MV4                  : 3481    4   7263    47.0 %   3502   125/256    2.2s    3.6s   0.41
 21 Lc0v0.30.0-dag+git.8260381-806638        : 3477    4   7340    46.1 %   3504   109/256    1.5s    3.5s   0.43
 22 RubiChess20221120(bmi2)                  : 3477    4   7150    46.3 %   3502   114/256    1.7s    3.5s   0.47
 23 Koivisto8.16                             : 3476    4   7190    46.1 %   3503   113/256    1.8s    3.6s   0.45
 24 Lc0v0.29.0-rc0-805992                    : 3474    4   7229    45.7 %   3505   113/256    1.8s    3.6s   0.42
 25 TheHuntsman1bmi2-MV4                     : 3473    4   7805    45.9 %   3501   109/256    1.6s    3.6s   0.32
 26 Ceres0.97RC3-784990                      : 3473    4   7358    45.5 %   3504   112/256    1.9s    3.6s   0.36
 27 Lc0v0.29.0-rc0-805874                    : 3471    4   7194    45.1 %   3505   109/256    1.7s    3.6s   0.41
 28 Minic3.312-Set2-MV4                      : 3471    4   7264    45.5 %   3503   119/256    2.2s    3.7s   0.39
 29 RubiChess20220813(avx2)                  : 3471    4   7132    45.4 %   3503   112/256    1.9s    3.6s   0.40
 30 Lc0v0.30.0-dag+git.8260381-MV4           : 3470    4   7291    45.1 %   3504   110/256    1.9s    3.7s   0.40
 31 Lc0v0.29.0-rc0-806638                    : 3470    4   7180    45.0 %   3505   109/256    1.8s    3.6s   0.41
 32 Lc0v0.29.0-rc1-806638                    : 3469    4   7174    44.9 %   3505   107/256    1.7s    3.6s   0.41
 33 Minic3.31-Set1-MV4                       : 3467    4   7219    44.9 %   3503   110/256    2.0s    3.7s   0.38
 34 Rebel16                                  : 3466    4   7133    44.6 %   3504   105/256    1.7s    3.7s   0.40
 35 Lc0v0.29.0-rc0-784968                    : 3462    4   7251    43.8 %   3505   106/256    2.1s    3.8s   0.33
 36 StingBlackHole                           : 3456    4   7508    43.1 %   3504    98/256    2.0s    3.8s   0.28
 37 Minic3.31-MV4                            : 3447    4   6852    41.8 %   3504    98/256    2.1s    3.9s   0.33
 38 Lc0v0.30.0-dag+git.c91bf77-784968        : 3437    4   6871    40.1 %   3506    86/256    1.9s    3.9s   0.30
 39 Wasp6.00-4MV                             : 3435    4   6762    40.0 %   3506    87/256    2.0s    4.0s   0.31
 40 PowerFritz18-MV4                         : 3435    4   6787    40.0 %   3505    97/256    2.5s    4.1s   0.27
 41 Seer2.6.0                                : 3434    4   6596    39.7 %   3506    84/256    1.7s    3.9s   0.34
 42 Rebel15.1a-MV4                           : 3430    4   6698    39.3 %   3506    85/256    2.0s    4.0s   0.28
 43 Halogen11-MV4                            : 3427    4   6618    38.7 %   3507    93/256    2.5s    4.1s   0.25
 44 Minic3.31                                : 3413    4   6452    36.7 %   3508    73/256    2.0s    4.1s   0.30
 45 Igel3.0.1064BMI2                         : 3392    4   6325    33.8 %   3509    61/256    2.0s    4.3s   0.23



MST1  : Mean solution time (solved positions only)
MST2  : Mean solution time (solved and unsolved positions)
RIndex: Score according to solution time ranking for each position

with these 256 positions

https://www.dropbox.com/s/lpg29zoyvh03dza/256.epd?dl=0

5"/pos., 30 threads of 16x3.5GHz and 3070ti Nvidia, MV4 means MultiPV=4, regards

[Eduard]

My engine Vulkan 021222 (based on Brainlearn 20.1) solves 114 of 120 positions in my test. Stockfish dev only manages 105 and Brainlearn 20.1 is even weaker. It's not the network.

Vulkan can be downloaded from my homepage under Solista News.

The fact of the matter is that Stockfish isn't getting any better. Maybe the testing environment has been good so far. Now it brings no more progress. The developers should think about changes. If you can't see any progress after 1.2 billion games after 8 months, you should change something.

[Eduard]

Stockfish in my EN-Test 2022:

Stockfish 15.txt Download:
https://filehorst.de/d/eruliwFg

(Other text files on my homepage)

Stockfish 15 manages 106 out of 120 positions on my PC. Stockfish dev manages 105 positions. There are some positions where Stockfish dev is better and some where Stockfish 15 is better. Here two examples:

All Analyses on Ryzen 3900X and 20 Threads. Hash 4 GB, all 3456men Syzygy.


[fen]4q1kr/p6p/1prQPppB/4n3/4P3/2P5/PP2B2P/R5K1 w - - 0 1[/fen]


Analysis by Stockfish 15 dev-avx2:

1.Qxe5 fxe5 2.Rf1 Rc8 3.Bd1 b5 4.Bb3 Rc4 5.Kg2 a6 6.Rf3 Qe7 7.Kg3 Qe8 8.a4 Qe7 9.Kg2 Qe8 10.h4 Qe7 11.Ba2 g5 12.hxg5 Qd6 13.Bb3 Qd8 14.Rf5 Qd6 15.Rf1 Qd8 16.axb5 axb5
+- (2.59) Depth: 29/39 00:00:01 22152kN, tb=68
White has a decisive advantage

Analysis by Stockfish dev 121222:

1.Qa3 Rxe6 2.Qxa7 Qe7 3.Qa8+ Qe8 4.Qb7 Qe7 5.Qc8+ Qe8 6.Qb7
= (0.00) Depth: 60/11 00:01:06 1143MN, tb=396050
The position is equal


[fen]rn3r1k/pn1p1ppq/bpp4p/7P/4N1Q1/6P1/PP3PB1/R1B1R1K1 w - - 0 1[/fen]


Analysis by Stockfish 15 dev-avx2:

21.b4 d5 22.Bb2 dxe4 23.Rxe4 f5 24.Qf4 Nd7 25.Re7 Nf6 26.Rd1 Nxh5 27.Qh4 Nf6 28.Rdd7 Rad8 29.Rxd8 Nxd8 30.Bxf6 Nf7 31.Rxa7 gxf6 32.Qxf6+ Qg7 33.Qxg7+ Kxg7 34.Rxa6 c5 35.Rxb6 cxb4 36.Rxb4 Rd8 37.Kf1 Rd2 38.a4 Nd6 39.Ke1 Rd3 40.Ke2 Ra3 41.Kd2 Kf6 42.Bd5 h5 43.Bb3 Ne4+ 44.Kc1 Ra1+ 45.Kb2 Rg1 46.a5 Nxf2 47.Kc2 Rc1+ 48.Kxc1 Nd3+ 49.Kd2 Nxb4 50.Kc3 Na6 51.Kc4 f4 52.gxf4 h4 53.Kb5 Nb8 54.Bd5
+- (2.34 ++) Depth: 35/58 00:00:31 617MN, tb=17499
White is clearly better

Analysis by Stockfish dev 121222:

21.Bg5 f5 22.Qf4 d5 23.Nf6 gxf6 24.Bxf6+ Rxf6 25.Re8+ Qg8 26.Rxg8+ Kxg8 27.Qc7 Rf7 28.Qc8+ Kg7 29.Qe6 Rf6 30.Qe7+ Rf7 31.Qe5+ Rf6 32.Re1 Nd7 33.Qe7+ Rf7 34.Qe6 Nf6 35.Qxf5 Raf8 36.Qg6+ Kh8 37.Qxh6+ Kg8 38.Re5 Rh7 39.Qe3 Nxh5 40.Re8 Rhf7 41.Rxf8+ Rxf8 42.Qe6+ Kg7 43.Qxc6 Nf6 44.g4
+- (1.96) Depth: 25/55 00:00:03 46437kN, tb=3
White is clearly better

Whether you use Stockfish 15 or Stockfish 15.1 shouldn't really matter.