Quick test of PhoenixStein vs Stockfish 12

[MMarco]

I made a quick test of PhoenixStein = jjosh's new net (author of LS nets for Allie and Lc0, and StockFiNN). On my weak GPU (for Leela), 1 core of Stockfish 12 using NNue (modern abrok compile) is sufficient to get a close match. LS 15.0 was used as a point of comparison.
Note that PhoenixStein is still in development, and stronger versions are likely to appear soon (I hope!).

Match conditions
100s + 1s.
GTX 1660 Ti and 1 core Ryzen 3750H.
Fixed openings played with both colors, 5-men syzygy.
SF: Threads=1, Hash=32.
Lc0: MiniBatchSize=128, MaxPrefetch=56, MaxCollisionEvents=56, SyzygyFastPlay=false.
Bench startpos (3sec). SF12 = 770 knps. PhoenixStein = 6.1 knps. LS 15.0 = 6.7 knps. 5-men syzygy.
Adjudication: 5 moves at 5cp for 5 moves in a row at move 30 (draws), 5 moves in a row at -600cp (resign).
Games: https://gofile.io/d/I2q7cP

First mini-match with silversuite (50 pos. still available here: https://en.chessbase.com/post/test-your ... ings-suite). I find this suite very well balanced strategically, but the draw rate is a bit high due the strenght of these engines. Lc0's did relatively better here.

Code: Select all

# PLAYER                     :  RATING  ERROR  PLAYED   (%)   CFS    W    D    L  D(%)
   1 Lc0 26.2 + LS 15.0         :    24.9   32.6     100  53.5    93   13   81    6  81.0
   2 Stockfish 12               :     0.0   ----     200  49.8    90   24  151   25  75.5
   3 Lc0 26.2 + PhoenixStein    :   -21.3   32.6     100  47.0   ---   12   70   18  70.0

Second mini-match was done using TCEC 18 Super Final openings (From Jereon Noomen chess blog: http://blogchess2016.blogspot.com/2020/ ... lable.html ), were more games were decisive, apparently adavantige Stockfish:

Code: Select all

   # PLAYER                     :  RATING  ERROR  PLAYED   (%)   CFS    W    D    L  D(%)
   1 Lc0 26.2 + LS 15.0         :     3.9   45.6     100  50.5    57   25   51   24  51.0
   2 Stockfish 12               :     0.0   ----     200  51.8    92   51  105   44  52.5
   3 Lc0 26.2 + PhoenixStein    :   -31.4   44.9     100  46.0   ---   19   54   27  54.0

Results combined: PhoenixStein is about 40 elo away from LS 15.0 (still much uncertainty remains)

Code: Select all

   # PLAYER                     :  RATING  ERROR  PLAYED   (%)   CFS    W    D    L  D(%)
   1 Lc0 26.2 + LS 15.0         :    14.7   34.1     200  52.0    80   38  132   30  66.0
   2 Stockfish 12               :     0.0   ----     400  50.8    96   75  256   69  64.0
   3 Lc0 26.2 + PhoenixStein    :   -25.8   29.2     200  46.5   ---   31  124   45  62.0

White advantage = 76.82 +/- 9.84
Draw rate (equal opponents) = 69.27 % +/- 2.36

Depth statistics: Phoenix reaches higher depth more quickly.

Code: Select all

Engine                      Time   Games  Moves  Lenght Sec/move  Depth   MIDG    EARLY   ENDG    LATE
Lc0 26.2 + PhoenixStein   7:36:54   200   11798  58.99    2.32    12.05   12.97 | 13.22 | 11.13 |  9.46
Lc0 26.2 + LS 15.0        7:57:41   200   12643  63,22    2.27    10.45   11.71 | 11.68 |  9.26 |  7.62
Stockfish 12             15:00:19   400   24432  61,08    2.21    26.90   24.11 | 24.27 | 27.39 | 38.84

A sample nice win by PhoenixStein were Stockfish is completely tied up despite being a rook up in the endgame:
[pgn][Date "2020.09.15"]
[Round "9"]
[White "Lc0 26.2 + PhoenixStein"]
[Black "Stockfish 12"]
[Result "1-0"]
[ECO "B99"]
[GameDuration "00:04:12"]
[GameEndTime "2020-09-15T01:16:29.122 Eastern time"]
[GameStartTime "2020-09-15T01:12:16.874 Eastern time"]
[Opening "Sicilian"]
[PlyCount "102"]
[Termination "adjudication"]
[TimeControl "100+1"]
[Variation "Najdorf, 7...Be7 Main line"]

1. e4 {book} c5 {book} 2. Nf3 {book} d6 {book} 3. d4 {book} cxd4 {book}
4. Nxd4 {book} Nf6 {book} 5. Nc3 {book} a6 {book} 6. Bg5 {book} e6 {book}
7. f4 {book} Be7 {book} 8. Qf3 {book} Qc7 {book} 9. O-O-O {book} Nbd7 {book}
10. g4 {book} b5 {book} 11. Bxf6 {book} Nxf6 {book} 12. g5 {book} Nd7 {book}
13. f5 {book} Nc5 {book} 14. f6 {book} gxf6 {book} 15. gxf6 {book} Bf8 {book}
16. Rg1 {book} h5 {-0.55/26 16s} 17. a3 {+0.33/10 2.3s} Rb8 {-0.37/21 1.3s}
18. Rg7 {+0.32/12 3.8s} b4 {+0.18/20 1.9s} 19. axb4 {+0.30/11 1.1s}
Rxb4 {0.00/24 2.4s} 20. Qe3 {+0.28/11 3.6s} Bd7 {0.00/24 4.9s}
21. b3 {+0.55/10 6.7s} Bxg7 {0.00/28 8.4s} 22. fxg7 {+0.38/22 1.8s}
Rg8 {0.00/26 1.2s} 23. Nf5 {+0.55/16 1.7s} exf5 {0.00/26 2.1s}
24. exf5+ {+0.12/22 3.9s} Kd8 {0.00/28 2.2s} 25. Nd5 {+0.10/22 2.3s}
Nxb3+ {0.00/25 1.8s} 26. Kb2 {+0.07/22 2.0s} Nc5+ {0.00/25 2.0s}
27. Nxb4 {+0.05/18 0.76s} Qb6 {0.00/28 1.7s} 28. Qd4 {+0.05/20 4.8s}
a5 {0.00/29 2.0s} 29. f6 {+0.07/17 1.4s} Kc7 {0.00/29 5.1s}
30. Kc1 {-0.05/15 7.7s} axb4 {0.00/29 1.5s} 31. Bc4 {-0.06/16 2.3s}
Be8 {0.00/29 2.2s} 32. Qf4 {-0.06/14 11s} Qc6 {+0.24/26 7.5s}
33. Bd5 {0.00/11 4.4s} Qa6 {+0.09/25 2.3s} 34. Kb1 {+0.02/10 6.0s}
Qe2 {0.00/28 4.2s} 35. Qd4 {+0.08/14 1.0s} Qg4 {+0.74/27 9.9s}
36. Qxg4 {+1.10/17 2.8s} hxg4 {-0.49/30 3.4s} 37. Re1 {+1.18/17 2.1s}
Kd8 {-0.38/29 1.4s} 38. Re7 {+1.32/16 3.0s} b3 {-0.20/30 1.6s}
39. cxb3 {+1.24/15 4.2s} Nd7 {-3.36/30 5.5s} 40. Rxd7+ {+1.38/15 2.4s}
Kxd7 {-3.60/34 5.8s} 41. Kc2 {+1.42/15 1.6s} Kc7 {-3.31/30 2.2s}
42. Kd3 {+1.77/16 3.9s} Kb6 {-4.05/33 2.6s} 43. Kd4 {+2.11/17 3.2s}
Kb5 {-4.54/35 4.7s} 44. Bc4+ {+2.76/16 3.6s} Kc6 {-4.48/31 1.4s}
45. b4 {+3.45/15 3.2s} Kb6 {-6.87/28 16s} 46. Kd5 {+6.23/14 3.8s}
Kc7 {-4.48/19 0.66s} 47. b5 {+8.19/13 2.3s} Bd7 {-6.59/21 2.0s}
48. b6+ {+10.27/13 2.3s} Kxb6 {-6.17/17 0.25s} 49. Kxd6 {+10.46/12 1.9s}
Rd8 {-8.67/20 4.0s} 50. Ke7 {+10.95/10 9.4s} Kc7 {-8.17/17 0.69s}
51. Bxf7 {+12.83/10 0.26s} g3 {-6.76/19 0.73s, White wins by adjudication} 1-0[/pgn]

[MMarco]

Quick changes! I tried the gambit suite by A. Silver. 25 positions. Games: https://gofile.io/d/Ql5BzR

Code: Select all

   # PLAYER                     :  RATING  ERROR  PLAYED   (%)   CFS    W    D    L  D(%)
   1 Stockfish 12               :    28.1   27.0     100  57.0    82   30   54   16  54.0
   2 Lc0 26.2 + PhoenixStein    :     0.0   45.5      50  46.0    84    9   28   13  56.0
   3 Lc0 26.2 + LS 15.0         :   -43.0   45.1      50  40.0   ---    7   26   17  52.0

White advantage = -14.36 +/- 25.95
Draw rate (equal opponents) = 55.54 % +/- 4.32

With the previous games, PhoenixStein is now within 25 elo from LS 15.0, and just a bit more from SF 12:

Code: Select all

   # PLAYER                     :  RATING  ERROR  PLAYED   (%)   CFS    W    D    L  D(%)
   1 Stockfish 12               :    26.0   13.4     500  52.0    58  105  310   85  62.0
   2 Lc0 26.2 + LS 15.0         :    23.1   21.0     250  49.6    88   45  158   47  63.2
   3 Lc0 26.2 + PhoenixStein    :     0.0   20.1     250  46.4   ---   40  152   58  60.8

White advantage = 58.16 +/- 8.94
Draw rate (equal opponents) = 64.64 % +/- 2.00