Stockfish 15 - Chess960 Testing Results

Discussion of computer chess matches and engine tournaments.

Moderators: hgm, Dann Corbit, Harvey Williamson

Modern Times
Posts: 3517
Joined: Thu Jun 07, 2012 11:02 pm

Stockfish 15 - Chess960 Testing Results

Post by Modern Times »

For the chess960 fans out there, my SF15 testing has been completed.

+46 Elo on this list which uses Bayeselo, same as fishtest. If you run the database through Ordo, it will tell you +56 Elo. Take your pick.

http://ccrl.chessdom.com/ccrl/404FRC/cg ... _length=30

Stockfish is so strong now that it really should not be playing Houdini 6 and Fire 8.1 due to the huge Elo difference, but I wanted to fill out the Top 10 table. Next time for Stockfsh 16, it will not play these two opponents.

I need to update the testing conditions text, the hash size in these tests has been 256MB for a long time now, same as the standard blitz list (for 1 thread).
Uri Blass
Posts: 10098
Joined: Thu Mar 09, 2006 12:37 am
Location: Tel-Aviv Israel

Re: Stockfish 15 - Chess960 Testing Results

Post by Uri Blass »

Interesting to note that Stockfish lost 2 games against Fire8.1 and 1 game against Houdini6 when stockfish did not lose a single game against
Fat Fritz 2 (in SF),Berserk 8.5, SlowChess Blitz 2.83,Revenge 2.0,RubiChess 20220223
Uri Blass
Posts: 10098
Joined: Thu Mar 09, 2006 12:37 am
Location: Tel-Aviv Israel

Re: Stockfish 15 - Chess960 Testing Results

Post by Uri Blass »

Here is the game that fire won with black

[pgn][Event "CCRL 40/2 FRC"]
[Site "CCRL"]
[Date "2022.04.19"]
[Round "300.3.27"]
[White "Stockfish 15"]
[Black "Fire 8.1"]
[Result "0-1"]
[Opening "BBRKQRNN"]
[PlyCount "113"]
[SetUp "1"]
[FEN "bbrkqrnn/pppppppp/8/8/8/8/PPPPPPPP/BBRKQRNN w KQkq - 0 1"]
[WhiteElo "3943"]
[BlackElo "3593"]

1. b3 {+0.30/24 3.0s} Nf6 {-0.19/19 7.5s} 2. e4 {+0.88/24 4.4s} c5 {-0.28/18
3.3s} 3. f4 {+0.87/20 0.94s} b5 {-0.22/15 1.1s} 4. Ng3 {+0.82/21 1.3s} d6
{-0.30/17 3.3s} 5. d3 {+1.04/21 2.9s} c4 {-0.17/15 0.89s} 6. N1e2 {+0.87/22
2.6s} Qd7 {-0.20/17 1.9s} 7. Bxf6 {+0.77/28 6.5s} exf6 {-0.08/15 0.87s} 8. d4
{+0.77/23 0.96s} f5 {+0.32/16 1.1s} 9. exf5 {+0.56/25 1.9s} Bxg2 {+0.42/15
0.97s} 10. Rg1 {+0.52/21 1.3s} Bd5 {+0.24/17 3.6s} 11. Nh5 {+0.64/20 1.4s} Bf3
{+0.30/16 1.1s} 12. Nxg7 {+0.76/23 1.6s} c3 {+0.29/19 2.0s} 13. Qf2 {+1.11/18
1.9s} Qc6 {+0.14/19 3.1s} 14. Rg3 {+0.93/20 1.7s} Bxe2+ {+0.15/21 2.8s} 15.
Kxe2 {+0.94/19 2.0s} b4 {+0.16/20 1.8s} 16. Rcg1 {+0.69/24 13s} a5 {+0.35/17
1.2s} 17. h4 {+1.03/20 7.1s} O-O-O {+0.55/16 1.8s} 18. Re3 {+0.00/23 9.4s} Rg8
{+0.67/17 2.0s} 19. Rg5 {+0.00/20 0.84s} h6 {+0.80/18 1.4s} 20. Rg4 {-0.70/24
7.8s} Ba7 {+0.89/17 0.39s} 21. Re7 {-0.83/20 0.76s} Rd7 {+0.89/15 0.001s} 22.
f6 {-1.29/23 6.3s} Ng6 {+0.91/19 3.4s} 23. Rxd7 {-1.49/21 1.9s} Qxd7 {+1.03/18
1.8s} 24. Qg2 {-1.32/19 1.4s} Kb8 {+1.03/16 0.001s} 25. h5 {-1.56/23 4.6s} Nf8
{+1.24/19 2.0s} 26. Nf5 {-1.74/20 0.90s} Rxg4 {+1.30/20 1.9s} 27. Qxg4
{-1.74/20 0.61s} Bxd4 {+1.59/18 1.7s} 28. Kf3 {-2.11/22 3.4s} Ne6 {+1.94/17
1.9s} 29. Kg3 {-3.01/21 2.5s} Bxf6 {+2.14/15 0s} 30. Qe2 {-2.78/18 0.36s} Nd4
{+2.65/21 2.1s} 31. Nxd4 {-1.97/20 0.34s} Bxd4 {+2.65/20 0s} 32. a4 {-2.73/22
1.8s} bxa3 {+2.97/24 6.4s} 33. b4 {-2.64/24 1.1s} axb4 {+3.05/24 2.8s} 34. Qc4
{-2.25/19 0.14s} Bc5 {+3.05/25 4.9s} 35. Ba2 {-3.17/23 0.70s} f5 {+3.06/25
3.7s} 36. Qg8+ {-3.01/18 0.16s} Ka7 {+2.28/25 6.5s} 37. Qf8 {-3.38/21 0.58s}
Qb5 {+2.28/24 0.012s} 38. Qf7+ {-3.40/19 0.22s} Kb6 {+2.35/23 3.5s} 39. Qf8
{-3.39/15 0.025s} Qc6 {+2.53/25 3.1s} 40. Qxf5 {-3.31/15 0.018s} Qh1 {+2.99/25
5.6s} 41. Qd3 {-7.88/26 23s} Qxh5 {+3.03/25 6.7s} 42. Bb3 {-6.26/24 5.8s} Qh1
{+3.83/21 2.1s} 43. Kg4 {-8.43/22 4.2s} Qa8 {+3.91/20 0.001s} 44. Bd5 {-9.62/26
16s} Qa7 {+4.20/21 2.1s} 45. Ba2 {-10.22/21 3.4s} b3 {+4.20/20 0s} 46. Bxb3
{-10.96/21 6.0s} a2 {+4.47/21 2.5s} 47. Bxa2 {-10.25/19 2.3s} Qxa2 {+4.86/20
2.2s} 48. Qe4 {-10.66/22 9.9s} Qf7 {+4.86/19 0s} 49. Qa4 {-13.67/21 7.7s} Qg6+
{+10.34/20 3.0s} 50. Kh3 {-13.68/21 1.5s} Qf5+ {+41.68/19 2.1s} 51. Kg3
{-16.89/22 5.5s} h5 {+50.18/26 2.2s} 52. Kh2 {-21.00/18 4.0s} Qg4 {+99.90/28
2.4s} 53. Qb3+ {-14.15/14 0.33s} Kc6 {+99.91/31 2.9s} 54. Qa4+ {-16.42/18 2.9s}
Kd5 {+99.92/31 3.0s} 55. Qa8+ {-18.43/17 2.1s} Kd4 {+99.93/32 4.7s} 56. Qh8+
{-19.48/17 1.3s} Ke4 {+99.94/34 2.6s} 57. f5 {-26.29/18 1.1s, Black wins by
adjudication} 0-1
[/pgn]

For some reason when I copy the pgn to arena gui it seems stockfish does not understand that castling of black is a move and analyze the position as black to move.
It does not happen with all engines that support FRC but only with part of them.

Dragon2.6.1 suggests castling but after going forward with castling in analysis mode inside arena it simply stop to respond and it seems not to understand the language of arena when some engines like Hermann2.8 have no problem with it.

I guess that the problem is not understanding the language
(maybe some interfaces treat castling as king take rook and some do it different when the best is if the engines understand all options).
Modern Times
Posts: 3517
Joined: Thu Jun 07, 2012 11:02 pm

Re: Stockfish 15 - Chess960 Testing Results

Post by Modern Times »

Copy it into the free version of Shredder GUI.

I did read somewhere that Cutechess has a very rare FRC bug, are you saying something is wrong with that game ?
Uri Blass
Posts: 10098
Joined: Thu Mar 09, 2006 12:37 am
Location: Tel-Aviv Israel

Re: Stockfish 15 - Chess960 Testing Results

Post by Uri Blass »

Is there a free version of shredder gui?

I do not see a free version here
https://www.shredderchess.com/windows/d ... er-13.html

I do not claim that there is a problem with cutechess but only that I cannot analyze the game in arena gui.
It may be possible to analyze it in cutechess or in a different free gui like banksia but I am not sure how to do it.
Modern Times
Posts: 3517
Joined: Thu Jun 07, 2012 11:02 pm

Re: Stockfish 15 - Chess960 Testing Results

Post by Modern Times »

Google Shredder classic, but it seems it is just a 30 day trial. I'm mistaken.
Modern Times
Posts: 3517
Joined: Thu Jun 07, 2012 11:02 pm

Re: Stockfish 15 - Chess960 Testing Results

Post by Modern Times »

Fire won two games as black. Strangely, when I open them in Shredder GUI it can't read them, it truncates the game. But Chessbase does open the full games, so I will check them there.



[Event "FRC SF15 gauntlet 3"]
[Site "CCRL"]
[Date "2022.04.19"]
[Round "69"]
[White "Stockfish 15 64-bit"]
[Black "Fire 8.1 64-bit"]
[Result "0-1"]
[FEN "qbrkbnrn/pppppppp/8/8/8/8/PPPPPPPP/QBRKBNRN w KQkq - 0 1"]
[GameDuration "00:05:23"]
[GameEndTime "2022-04-19T22:47:49.003 GMT Summer Time"]
[GameStartTime "2022-04-19T22:42:25.378 GMT Summer Time"]
[PlyCount "117"]
[SetUp "1"]
[Termination "adjudication"]
[TimeControl "40/100"]
[Variant "fischerandom"]

1. d4 {+0.48/21 3.9s} f6 {-0.30/17 4.9s} 2. c4 {+0.93/23 5.1s}
c5 {-0.31/17 2.4s} 3. d5 {+1.14/19 0.85s} e6 {-0.16/15 1.1s}
4. Nhg3 {+1.12/20 1.2s} Nfg6 {-0.25/19 5.5s} 5. h4 {+1.18/23 2.8s}
Nf7 {-0.16/16 2.0s} 6. h5 {+1.34/21 1.0s} Ne7 {-0.29/17 4.2s}
7. O-O-O {+1.50/21 1.1s} exd5 {-0.18/16 1.0s} 8. cxd5 {+1.40/23 2.1s}
a5 {-0.12/18 3.9s} 9. a4 {+1.32/21 2.0s} g6 {-0.19/16 2.1s}
10. Ne3 {+1.43/23 7.2s} Qa6 {+0.12/14 1.1s} 11. Qa2 {+1.60/20 1.0s}
Bf4 {-0.34/18 4.8s} 12. hxg6 {+1.10/21 4.1s} hxg6 {-0.33/17 1.6s}
13. Rh1 {+1.42/25 6.7s} f5 {-0.27/18 2.0s} 14. Bc3 {+1.25/25 3.6s}
d6 {-0.32/18 4.4s} 15. Rh7 {+1.37/20 2.4s} Rf8 {-0.21/18 1.2s}
16. Bc2 {+1.31/25 5.7s} b5 {-0.08/17 1.1s} 17. axb5 {+1.62/20 1.2s}
Bxb5 {-0.08/17 2.0s} 18. Nf1 {+0.92/23 4.0s} a4 {-0.06/18 1.9s}
19. g3 {+0.96/24 2.6s} Bxe3+ {-0.03/19 1.6s} 20. Nxe3 {+0.66/26 2.4s}
Ng5 {+0.04/19 1.2s} 21. Rg7 {+1.02/23 1.3s} Ne4 {+0.06/19 2.0s}
22. Rh1 {+0.63/30 8.3s} Nxc3 {+0.18/18 1.4s} 23. bxc3 {+0.57/27 2.8s}
a3 {-0.06/22 3.9s} 24. Rhh7 {+0.61/24 1.1s} Re8 {+0.20/19 1.2s}
25. Bd3 {0.00/29 6.6s} c4 {+0.42/18 2.2s} 26. Bb1 {0.00/28 0.64s}
Ra8 {+0.33/20 3.1s} 27. Nc2 {0.00/24 1.5s} Qb6 {-0.02/23 8.5s}
28. Nd4 {0.00/20 0.81s} Bd7 {+0.48/17 1.1s} 29. g4 {-0.79/23 5.0s}
Qa5 {+0.64/17 1.4s} 30. Kd2 {-0.43/18 0.52s} Qxd5 {+0.33/20 3.6s}
31. gxf5 {-0.95/23 3.7s} gxf5 {+0.56/20 1.4s} 32. Bc2 {-0.93/20 1.7s}
Qc5 {+0.60/19 1.7s} 33. e3 {-1.16/18 0.77s} Ra6 {+0.61/19 1.7s}
34. f4 {-0.81/21 1.9s} d5 {+0.74/19 3.1s} 35. Bd1 {-0.86/16 0.21s}
Rb6 {+0.77/19 1.7s} 36. Bc2 {-1.07/24 1.0s} Qa5 {+0.49/24 3.2s}
37. Rf7 {-0.87/21 0.32s} Rb2 {+0.49/23 1.7s} 38. Qa1 {-1.50/20 0.54s}
Qa8 {+0.49/24 1.8s} 39. Ke2 {0.00/16 0.11s} Qa7 {+1.15/20 1.3s}
40. Kf3 {-0.07/12 0.022s} a2 {+2.31/18 1.7s} 41. Rh6 {-4.64/23 23s}
Qa3 {+3.51/18 2.4s} 42. Rd6 {-5.92/23 18s} Rxc2 {+4.46/20 2.1s}
43. Nxc2 {-6.51/23 5.1s} Qxd6 {+4.66/21 2.5s} 44. Qxa2 {-6.57/23 1.7s}
Rg8 {+4.82/20 2.3s} 45. Qa8+ {-6.48/21 1.4s} Bc8 {+5.04/20 0s}
46. Ke2 {-7.11/26 13s} Qh6 {+6.01/20 2.0s} 47. Kd2 {-7.27/21 1.4s}
Rg2+ {+6.01/19 0.002s} 48. Kc1 {-9.06/22 3.7s} Rg1+ {+7.57/22 3.5s}
49. Kb2 {-11.11/20 3.6s} Qb6+ {+7.57/22 0s} 50. Nb4 {-11.18/21 2.9s}
Qxe3 {+8.67/23 3.5s} 51. Rf8+ {-12.84/20 7.3s} Kd7 {+10.01/21 2.6s}
52. Qa4+ {-11.95/16 1.0s} Ke6 {+10.01/21 0s} 53. Qc2 {-15.41/20 5.2s}
d4 {+25.75/19 3.0s} 54. Rd8 {-19.94/18 3.8s} dxc3+ {+99.93/29 2.9s}
55. Qxc3 {-16.90/16 1.1s} Rb1+ {+M35/28 0.75s} 56. Kxb1 {-23.56/19 1.3s}
Qxc3 {+M25/32 1.4s} 57. Nc2 {-31.65/17 2.1s} Bd7 {+M23/30 0.037s}
58. Ra8 {-17.72/16 1.5s} Nd5 {+M21/28 0.008s}
59. Ra6+ {-14.52/12 0.23s, Black wins by adjudication} 0-1

[Event "FRC SF15 gauntlet 3"]
[Site "CCRL"]
[Date "2022.04.19"]
[Round "66"]
[White "Stockfish 15 64-bit"]
[Black "Fire 8.1 64-bit"]
[Result "0-1"]
[FEN "bbrkqrnn/pppppppp/8/8/8/8/PPPPPPPP/BBRKQRNN w KQkq - 0 1"]
[GameDuration "00:05:29"]
[GameEndTime "2022-04-19T22:37:07.434 GMT Summer Time"]
[GameStartTime "2022-04-19T22:31:38.308 GMT Summer Time"]
[PlyCount "113"]
[SetUp "1"]
[Termination "adjudication"]
[TimeControl "40/100"]
[Variant "fischerandom"]

1. b3 {+0.30/24 3.0s} Nf6 {-0.19/19 7.5s} 2. e4 {+0.88/24 4.4s}
c5 {-0.28/18 3.3s} 3. f4 {+0.87/20 0.94s} b5 {-0.22/15 1.1s}
4. Ng3 {+0.82/21 1.3s} d6 {-0.30/17 3.3s} 5. d3 {+1.04/21 2.9s}
c4 {-0.17/15 0.89s} 6. N1e2 {+0.87/22 2.6s} Qd7 {-0.20/17 1.9s}
7. Bxf6 {+0.77/28 6.5s} exf6 {-0.08/15 0.87s} 8. d4 {+0.77/23 0.96s}
f5 {+0.32/16 1.1s} 9. exf5 {+0.56/25 1.9s} Bxg2 {+0.42/15 0.97s}
10. Rg1 {+0.52/21 1.3s} Bd5 {+0.24/17 3.6s} 11. Nh5 {+0.64/20 1.4s}
Bf3 {+0.30/16 1.1s} 12. Nxg7 {+0.76/23 1.6s} c3 {+0.29/19 2.0s}
13. Qf2 {+1.11/18 1.9s} Qc6 {+0.14/19 3.1s} 14. Rg3 {+0.93/20 1.7s}
Bxe2+ {+0.15/21 2.8s} 15. Kxe2 {+0.94/19 2.0s} b4 {+0.16/20 1.8s}
16. Rcg1 {+0.69/24 13s} a5 {+0.35/17 1.2s} 17. h4 {+1.03/20 7.1s}
O-O-O {+0.55/16 1.8s} 18. Re3 {0.00/23 9.4s} Rg8 {+0.67/17 2.0s}
19. Rg5 {0.00/20 0.84s} h6 {+0.80/18 1.4s} 20. Rg4 {-0.70/24 7.8s}
Ba7 {+0.89/17 0.39s} 21. Re7 {-0.83/20 0.76s} Rd7 {+0.89/15 0.001s}
22. f6 {-1.29/23 6.3s} Ng6 {+0.91/19 3.4s} 23. Rxd7 {-1.49/21 1.9s}
Qxd7 {+1.03/18 1.8s} 24. Qg2 {-1.32/19 1.4s} Kb8 {+1.03/16 0.001s}
25. h5 {-1.56/23 4.6s} Nf8 {+1.24/19 2.0s} 26. Nf5 {-1.74/20 0.90s}
Rxg4 {+1.30/20 1.9s} 27. Qxg4 {-1.74/20 0.61s} Bxd4 {+1.59/18 1.7s}
28. Kf3 {-2.11/22 3.4s} Ne6 {+1.94/17 1.9s} 29. Kg3 {-3.01/21 2.5s}
Bxf6 {+2.14/15 0s} 30. Qe2 {-2.78/18 0.36s} Nd4 {+2.65/21 2.1s}
31. Nxd4 {-1.97/20 0.34s} Bxd4 {+2.65/20 0s} 32. a4 {-2.73/22 1.8s}
bxa3 {+2.97/24 6.4s} 33. b4 {-2.64/24 1.1s} axb4 {+3.05/24 2.8s}
34. Qc4 {-2.25/19 0.14s} Bc5 {+3.05/25 4.9s} 35. Ba2 {-3.17/23 0.70s}
f5 {+3.06/25 3.7s} 36. Qg8+ {-3.01/18 0.16s} Ka7 {+2.28/25 6.5s}
37. Qf8 {-3.38/21 0.58s} Qb5 {+2.28/24 0.012s} 38. Qf7+ {-3.40/19 0.22s}
Kb6 {+2.35/23 3.5s} 39. Qf8 {-3.39/15 0.025s} Qc6 {+2.53/25 3.1s}
40. Qxf5 {-3.31/15 0.018s} Qh1 {+2.99/25 5.6s} 41. Qd3 {-7.88/26 23s}
Qxh5 {+3.03/25 6.7s} 42. Bb3 {-6.26/24 5.8s} Qh1 {+3.83/21 2.1s}
43. Kg4 {-8.43/22 4.2s} Qa8 {+3.91/20 0.001s} 44. Bd5 {-9.62/26 16s}
Qa7 {+4.20/21 2.1s} 45. Ba2 {-10.22/21 3.4s} b3 {+4.20/20 0s}
46. Bxb3 {-10.96/21 6.0s} a2 {+4.47/21 2.5s} 47. Bxa2 {-10.25/19 2.3s}
Qxa2 {+4.86/20 2.2s} 48. Qe4 {-10.66/22 9.9s} Qf7 {+4.86/19 0s}
49. Qa4 {-13.67/21 7.7s} Qg6+ {+10.34/20 3.0s} 50. Kh3 {-13.68/21 1.5s}
Qf5+ {+41.68/19 2.1s} 51. Kg3 {-16.89/22 5.5s} h5 {+50.18/26 2.2s}
52. Kh2 {-21.00/18 4.0s} Qg4 {+99.90/28 2.4s} 53. Qb3+ {-14.15/14 0.33s}
Kc6 {+99.91/31 2.9s} 54. Qa4+ {-16.42/18 2.9s} Kd5 {+99.92/31 3.0s}
55. Qa8+ {-18.43/17 2.1s} Kd4 {+99.93/32 4.7s} 56. Qh8+ {-19.48/17 1.3s}
Ke4 {+99.94/34 2.6s} 57. f5 {-26.29/18 1.1s, Black wins by adjudication} 0-1
Modern Times
Posts: 3517
Joined: Thu Jun 07, 2012 11:02 pm

Re: Stockfish 15 - Chess960 Testing Results

Post by Modern Times »

I played through the first of these two games in Fritz 17 GUI. It all goes wrong for Stockfish on move 40, where it seems to have got into time trouble with the repeating time control on move 40 and then on move 41 it realises it is a disaster

40. Kf3 {-0.07/12 0.022s} a2 {+2.31/18 1.7s} 41. Rh6 {-4.64/23 23s}

However prior to that it does seem that it was genuinely being out-played by Fire, and over-evaluating its own position.

Maybe the Move Overhead UCI parameter (default 10) needs increasing slightly next time I run it, however this loss would probably still have occurred.