Quick test of PhoenixStein vs Stockfish 12

Discussion of computer chess matches and engine tournaments.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
MMarco
Posts: 120
Joined: Sat Apr 11, 2020 11:09 pm
Full name: Marc-O Moisan-Plante

Quick test of PhoenixStein vs Stockfish 12

Post by MMarco » Wed Sep 16, 2020 1:35 pm

I made a quick test of PhoenixStein = jjosh's new net (author of LS nets for Allie and Lc0, and StockFiNN). On my weak GPU (for Leela), 1 core of Stockfish 12 using NNue (modern abrok compile) is sufficient to get a close match. LS 15.0 was used as a point of comparison.
Note that PhoenixStein is still in development, and stronger versions are likely to appear soon (I hope!).

Match conditions
100s + 1s.
GTX 1660 Ti and 1 core Ryzen 3750H.
Fixed openings played with both colors, 5-men syzygy.
SF: Threads=1, Hash=32.
Lc0: MiniBatchSize=128, MaxPrefetch=56, MaxCollisionEvents=56, SyzygyFastPlay=false.
Bench startpos (3sec). SF12 = 770 knps. PhoenixStein = 6.1 knps. LS 15.0 = 6.7 knps. 5-men syzygy.
Adjudication: 5 moves at 5cp for 5 moves in a row at move 30 (draws), 5 moves in a row at -600cp (resign).
Games: https://gofile.io/d/I2q7cP

First mini-match with silversuite (50 pos. still available here: https://en.chessbase.com/post/test-your ... ings-suite). I find this suite very well balanced strategically, but the draw rate is a bit high due the strenght of these engines. Lc0's did relatively better here.

Code: Select all

# PLAYER                     :  RATING  ERROR  PLAYED   (%)   CFS    W    D    L  D(%)
   1 Lc0 26.2 + LS 15.0         :    24.9   32.6     100  53.5    93   13   81    6  81.0
   2 Stockfish 12               :     0.0   ----     200  49.8    90   24  151   25  75.5
   3 Lc0 26.2 + PhoenixStein    :   -21.3   32.6     100  47.0   ---   12   70   18  70.0
Second mini-match was done using TCEC 18 Super Final openings (From Jereon Noomen chess blog: http://blogchess2016.blogspot.com/2020/ ... lable.html ), were more games were decisive, apparently adavantige Stockfish:

Code: Select all

   # PLAYER                     :  RATING  ERROR  PLAYED   (%)   CFS    W    D    L  D(%)
   1 Lc0 26.2 + LS 15.0         :     3.9   45.6     100  50.5    57   25   51   24  51.0
   2 Stockfish 12               :     0.0   ----     200  51.8    92   51  105   44  52.5
   3 Lc0 26.2 + PhoenixStein    :   -31.4   44.9     100  46.0   ---   19   54   27  54.0
Results combined: PhoenixStein is about 40 elo away from LS 15.0 (still much uncertainty remains)

Code: Select all

   # PLAYER                     :  RATING  ERROR  PLAYED   (%)   CFS    W    D    L  D(%)
   1 Lc0 26.2 + LS 15.0         :    14.7   34.1     200  52.0    80   38  132   30  66.0
   2 Stockfish 12               :     0.0   ----     400  50.8    96   75  256   69  64.0
   3 Lc0 26.2 + PhoenixStein    :   -25.8   29.2     200  46.5   ---   31  124   45  62.0

White advantage = 76.82 +/- 9.84
Draw rate (equal opponents) = 69.27 % +/- 2.36
Depth statistics: Phoenix reaches higher depth more quickly.

Code: Select all

Engine                      Time   Games  Moves  Lenght Sec/move  Depth   MIDG    EARLY   ENDG    LATE
Lc0 26.2 + PhoenixStein   7:36:54   200   11798  58.99    2.32    12.05   12.97 | 13.22 | 11.13 |  9.46
Lc0 26.2 + LS 15.0        7:57:41   200   12643  63,22    2.27    10.45   11.71 | 11.68 |  9.26 |  7.62
Stockfish 12             15:00:19   400   24432  61,08    2.21    26.90   24.11 | 24.27 | 27.39 | 38.84
A sample nice win by PhoenixStein were Stockfish is completely tied up despite being a rook up in the endgame:

MMarco
Posts: 120
Joined: Sat Apr 11, 2020 11:09 pm
Full name: Marc-O Moisan-Plante

Re: Quick test of PhoenixStein vs Stockfish 12

Post by MMarco » Wed Sep 16, 2020 10:07 pm

Quick changes! I tried the gambit suite by A. Silver. 25 positions. Games: https://gofile.io/d/Ql5BzR

Code: Select all

   # PLAYER                     :  RATING  ERROR  PLAYED   (%)   CFS    W    D    L  D(%)
   1 Stockfish 12               :    28.1   27.0     100  57.0    82   30   54   16  54.0
   2 Lc0 26.2 + PhoenixStein    :     0.0   45.5      50  46.0    84    9   28   13  56.0
   3 Lc0 26.2 + LS 15.0         :   -43.0   45.1      50  40.0   ---    7   26   17  52.0

White advantage = -14.36 +/- 25.95
Draw rate (equal opponents) = 55.54 % +/- 4.32
With the previous games, PhoenixStein is now within 25 elo from LS 15.0, and just a bit more from SF 12:

Code: Select all

   # PLAYER                     :  RATING  ERROR  PLAYED   (%)   CFS    W    D    L  D(%)
   1 Stockfish 12               :    26.0   13.4     500  52.0    58  105  310   85  62.0
   2 Lc0 26.2 + LS 15.0         :    23.1   21.0     250  49.6    88   45  158   47  63.2
   3 Lc0 26.2 + PhoenixStein    :     0.0   20.1     250  46.4   ---   40  152   58  60.8

White advantage = 58.16 +/- 8.94
Draw rate (equal opponents) = 64.64 % +/- 2.00

Post Reply