Lc0 J92-100, SFNNue, Allie Dev round-robin

Discussion of computer chess matches and engine tournaments.

Moderator: Ras

MMarco
Posts: 212
Joined: Sun Apr 12, 2020 1:09 am
Full name: Marc-O Moisan-Plante

Lc0 J92-100, SFNNue, Allie Dev round-robin

Post by MMarco »

I ran a tourney over the last week or so with some of the latest engines and nets. Latest available Allie dev seems to be about on par with the best SFNNue (1 CPU) and CFish 11a (8 CPU) in my match conditions. Lc0 26.1 with cuda 11 and cudnn 8.0.2 is about 30% faster than the standard released 26.1 for large nets on my system. So these conditions were somewhat advantageous speedwise for Lc0 (though time control was not long for a big net like J92-100) compared to the usual standard at TCEC or with Leela Ratio, but they kept the tournament interesting. Allie's compile used cuda 10 (there is no publically available cuda 11 compile of it to my knowledge). Net SV-1705 was rejected in fishtest against nn-82215d0fd0df by 3 elo over thousand of games, but did slightly better here although within the error margin.

Tournament: 1 CPU SFNNue vs Lc0 RTX 2060
Time control: 100s + 1s
Hardware: Ryzen 9 4900H, RTX 2060, 5-men syzygy
Openings: TCEC SuFi 2016 (50 pos.) + 6-ply openings (50 pos.). Same as AlphaZero-Stockfish simulated rematch at CCC 2020/08
Allie: MaxBatchSize=120, UseFP16=true, UseCustomWinograd=true, from Allie's discord
CFish 11a: Threads=8, Hash = 512, zen2_avx2 compile (official release)
Stockfishes: Threads=1, Hash=64, avx2_Ryzen compile by ChessMan (Outskirts forum)
Lc0 J92-100: cuda 11 + cudnn 8.0.2 (Borg's compile on Lc0 discord), --minibatch-size = 112, --max-prefetch = 70, --moves-left-max-effect=0.100000, --moves-left-threshold=0.862976, --moves-left-slope=0.007000, --max-out-of-order-evals-factor=2.4
Bench startpos (3.3s). SFNNue 170820: Pure = 1400 knps (70% of classical). Lc0 J92-100 = 4.3 knps. Allie = 51 knps. CFish 11a 8CPU = 15 000 knps.
Adjudication: resign movecount=5 score=600, draw movenumber=35 movecount=5 score=5
Games: https://gofile.io/d/6HsZsl

Code: Select all

   # PLAYER                       :  RATING  ERROR  PLAYED    (%)   CFS    W    D    L   D(%)
   1 Lc0 26.1 + J92-100 (c11)     :    21.1    9.7     800  52.56    96  128  585   87  73.13
   2 SF 170820 SV-1705            :     6.4    9.9     800  50.00    62  103  594  103  74.25
   3 Allie 120820 + LS 15.0       :     4.3    9.2     800  49.63    69  110  574  116  71.75
   4 CFish 11a 8CPU               :     0.4    9.7     800  48.94    52  112  559  129  69.88
   5 SF 170820 nn-82215d0fd0df    :     0.0    9.5     800  48.88   ---   96  590  114  73.75

White advantage = 49.99 +/- 4.20
Draw rate (equal opponents) = 75.68 % +/- 0.95
Head to head statistics:

Code: Select all

1) Lc0 26.1 + J92-100 (c11)  21.1 :    800 (+128,=585,-87),  52.6 %

   vs.                              :  games (   +,   =,  -),   (%) :    Diff,    SD, CFS (%)
   SF 170820 SV-1705                :    200 (  35, 146, 19),  54.0 :   +14.7,   8.6,   95.6
   Allie 120820 + LS 15.0           :    200 (  22, 160, 18),  51.0 :   +16.8,   7.2,   99.0
   CFish 11a 8CPU                   :    200 (  37, 137, 26),  52.8 :   +20.8,   7.7,   99.6
   SF 170820 nn-82215d0fd0df        :    200 (  34, 142, 24),  52.5 :   +21.1,   7.4,   99.8

2) SF 170820 SV-1705          6.4 :    800 (+103,=594,-103),  50.0 %

   vs.                              :  games (   +,   =,   -),   (%) :    Diff,    SD, CFS (%)
   Lc0 26.1 + J92-100 (c11)         :    200 (  19, 146,  35),  46.0 :   -14.7,   8.6,    4.4
   Allie 120820 + LS 15.0           :    200 (  34, 142,  24),  52.5 :    +2.1,   7.3,   61.6
   CFish 11a 8CPU                   :    200 (  32, 142,  26),  51.5 :    +6.1,   7.7,   78.5
   SF 170820 nn-82215d0fd0df        :    200 (  18, 164,  18),  50.0 :    +6.4,   7.8,   79.4

3) Allie 120820 + LS 15.0     4.3 :    800 (+110,=574,-116),  49.6 %

   vs.                              :  games (   +,   =,   -),   (%) :    Diff,    SD, CFS (%)
   Lc0 26.1 + J92-100 (c11)         :    200 (  18, 160,  22),  49.0 :   -16.8,   7.2,    1.0
   SF 170820 SV-1705                :    200 (  24, 142,  34),  47.5 :    -2.1,   7.3,   38.4
   CFish 11a 8CPU                   :    200 (  32, 134,  34),  49.5 :    +3.9,   8.0,   68.9
   SF 170820 nn-82215d0fd0df        :    200 (  36, 138,  26),  52.5 :    +4.3,   7.8,   70.9

4) CFish 11a 8CPU             0.4 :    800 (+112,=559,-129),  48.9 %

   vs.                              :  games (   +,   =,   -),   (%) :    Diff,    SD, CFS (%)
   Lc0 26.1 + J92-100 (c11)         :    200 (  26, 137,  37),  47.3 :   -20.8,   7.7,    0.4
   SF 170820 SV-1705                :    200 (  26, 142,  32),  48.5 :    -6.1,   7.7,   21.5
   Allie 120820 + LS 15.0           :    200 (  34, 134,  32),  50.5 :    -3.9,   8.0,   31.1
   SF 170820 nn-82215d0fd0df        :    200 (  26, 146,  28),  49.5 :    +0.4,   7.7,   51.9

5) SF 170820 nn-82215d0fd0df  0.0 :    800 (+96,=590,-114),  48.9 %

   vs.                              :  games (  +,   =,   -),   (%) :    Diff,    SD, CFS (%)
   Lc0 26.1 + J92-100 (c11)         :    200 ( 24, 142,  34),  47.5 :   -21.1,   7.4,    0.2
   SF 170820 SV-1705                :    200 ( 18, 164,  18),  50.0 :    -6.4,   7.8,   20.6
   Allie 120820 + LS 15.0           :    200 ( 26, 138,  36),  47.5 :    -4.3,   7.8,   29.1
   CFish 11a 8CPU                   :    200 ( 28, 146,  26),  50.5 :    -0.4,   7.7,   48.1
Top
MMarco
Posts: 212
Joined: Sun Apr 12, 2020 1:09 am
Full name: Marc-O Moisan-Plante

Re: Lc0 J92-100, SFNNue, Allie Dev round-robin

Post by MMarco »

Round-Robin: SF NNue 1CPU, Lc0, Allie, CFish 11a 8CPU
Time control: 100s + 1s
Hardware: Ryzen 4900H, RTX 2600
Openings: TCEC SuFi 2016 (50 pos.) + 6-ply openings (50 pos.) as in Alphazero-Stockfish simulated rematch at CCC 2020/08
CFish 11a: Threads=8, Hash = 512, zen2_avx2
SF 170820: Threads=1, Hash=64, avx2_Ryzen (Chessman)
Allie 120820: MaxBatchSize=120, UseFP16=true, UseCustomWinograd=true
Lc0 26.2 SV-5300: backend-opts=custom_winograd=false, minibatch-size=120, max-prefetch=54, max-out-of-order-evals-factor=2, syzygy-fast-play=false
Lc0 26.1 J92-100: cuda 11 + cudnn 8.0.2 (Borg), minibatch-size=112, max-prefetch=70, max-out-of-order-evals-factor=2.4, moves-left-max-effect=0.1, moves-left-threshold=0.862976, moves-left-slope=0.007, syzygy-fast-play=false
Bench startpos (3.3s): SF 170820 NNUE = 1400 knps. CFish 11a 8CPU = 15000 knps. Lc0 J92-100 = 4.3 knps. Lc0 SV-5300 = 90 knps. Allie=50 knps.
TBs and adj.: Syzygy 5-men, resign movecount=5 score=600, draw movenumber=35 movecount=5 score=5
Comment: 1 CPU SF NNue on par with the 10x128 net here.
Games: https://gofile.io/d/FePTwM
Head-to-head statistics: https://gofile.io/d/goeYLw

Code: Select all

   # PLAYER                      :  RATING  ERROR  PLAYED    (%)   CFS    W    D    L   D(%)
   1 Lc0 26.1 + J92-100 MLH      :    22.4    9.0    1000  52.85    99  158  741  101  74.10
   2 SF 170820 + SV-1705 1CPU    :     5.4    8.9    1000  50.00    71  125  750  125  75.00
   3 Allie 120820 + LS 15.0      :     1.8    7.8    1000  49.40    50  128  732  140  73.20
   4 Lc0 26.2 + SV-5300          :     1.8    9.3    1000  49.40    55  113  762  125  76.20
   5 CFish 11a 8CPU              :     0.9    9.1    1000  49.25    55  143  699  158  69.90
   6 SF 170820 + SV-2257 1CPU    :     0.0    9.0    1000  49.10   ---  120  742  138  74.20

White advantage = 49.09 +/- 3.10
Draw rate (equal opponents) = 77.00 % +/- 0.80
Top
MMarco
Posts: 212
Joined: Sun Apr 12, 2020 1:09 am
Full name: Marc-O Moisan-Plante

Re: Lc0 J92-100, SFNNue, Allie Dev round-robin

Post by MMarco »

CFish 050920 + SV-1739 added.

CFish NNue: threads=1, hash=64, avx2_Ryzen (compile by Chessman)
Bench startpos (3.3s): CFish 050920 = 1600 knps.
Comment: CFish at the top, but narrowly lose match against J92-100.
Games: https://gofile.io/d/tWAFbD

Code: Select all

   # PLAYER                      :  RATING  ERROR  PLAYED    (%)   CFS    W    D    L   D(%)
   1 CFish 050920 + SV-1739      :    32.7    9.0    1200  53.46    81  182  919   99  76.58
   2 Lc0 26.1 + J92-100 (c11)    :    26.3    8.4    1200  52.42    99  181  896  123  74.67
   3 SF 170820 + SV-1705         :     8.4    9.1    1200  49.50    67  138  912  150  76.00
   4 Allie 120820 + LS 15.0      :     5.6    7.5    1200  49.04    55  153  871  176  72.58
   5 Lc0 26.2 + SV-5300          :     4.9    7.5    1200  48.92    64  132  910  158  75.83
   6 CFish 11a 8CPU              :     2.6    7.8    1200  48.54    65  154  857  189  71.42
   7 SF 170820 + SV-2257         :     0.0    8.0    1200  48.13   ---  128  899  173  74.92

White advantage = 48.84 +/- 2.88
Draw rate (equal opponents) = 78.14 % +/- 0.64
Head-to-head statistics:

Code: Select all

1) CFish 050920 + SV-1739   32.7 :   1200 (+182,=919,-99),  53.5 %

   vs.                             :  games (   +,   =,  -),   (%)
   Lc0 26.1 + J92-100 (c11)        :    200 (  22, 155, 23),  49.8
   SF 170820 + SV-1705             :    200 (  25, 162, 13),  53.0
   Allie 120820 + LS 15.0          :    200 (  36, 139, 25),  52.8
   Lc0 26.2 + SV-5300              :    200 (  33, 148, 19),  53.5
   CFish 11a 8CPU                  :    200 (  31, 158, 11),  55.0
   SF 170820 + SV-2257             :    200 (  35, 157,  8),  56.8

2) Lc0 26.1 + J92-100 (c11) 26.3 :   1200 (+181,=896,-123),  52.4 %

   vs.                             :  games (   +,   =,   -),   (%)
   CFish 050920 + SV-1739          :    200 (  23, 155,  22),  50.3
   SF 170820 + SV-1705             :    200 (  35, 146,  19),  54.0
   Allie 120820 + LS 15.0          :    200 (  22, 160,  18),  51.0
   Lc0 26.2 + SV-5300              :    200 (  30, 156,  14),  54.0
   CFish 11a 8CPU                  :    200 (  37, 137,  26),  52.8
   SF 170820 + SV-2257             :    200 (  34, 142,  24),  52.5

3) SF 170820 + SV-1705       8.4 :   1200 (+138,=912,-150),  49.5 %

   vs.                             :  games (   +,   =,   -),   (%)
   CFish 050920 + SV-1739          :    200 (  13, 162,  25),  47.0
   Lc0 26.1 + J92-100 (c11)        :    200 (  19, 146,  35),  46.0
   Allie 120820 + LS 15.0          :    200 (  34, 142,  24),  52.5
   Lc0 26.2 + SV-5300              :    200 (  22, 156,  22),  50.0
   CFish 11a 8CPU                  :    200 (  32, 142,  26),  51.5
   SF 170820 + SV-2257             :    200 (  18, 164,  18),  50.0

4) Allie 120820 + LS 15.0    5.6 :   1200 (+153,=871,-176),  49.0 %

   vs.                             :  games (   +,   =,   -),   (%)
   CFish 050920 + SV-1739          :    200 (  25, 139,  36),  47.3
   Lc0 26.1 + J92-100 (c11)        :    200 (  18, 160,  22),  49.0
   SF 170820 + SV-1705             :    200 (  24, 142,  34),  47.5
   Lc0 26.2 + SV-5300              :    200 (  18, 158,  24),  48.5
   CFish 11a 8CPU                  :    200 (  32, 134,  34),  49.5
   SF 170820 + SV-2257             :    200 (  36, 138,  26),  52.5

5) Lc0 26.2 + SV-5300        4.9 :   1200 (+132,=910,-158),  48.9 %

   vs.                             :  games (   +,   =,   -),   (%)
   CFish 050920 + SV-1739          :    200 (  19, 148,  33),  46.5
   Lc0 26.1 + J92-100 (c11)        :    200 (  14, 156,  30),  46.0
   SF 170820 + SV-1705             :    200 (  22, 156,  22),  50.0
   Allie 120820 + LS 15.0          :    200 (  24, 158,  18),  51.5
   CFish 11a 8CPU                  :    200 (  29, 140,  31),  49.5
   SF 170820 + SV-2257             :    200 (  24, 152,  24),  50.0

6) CFish 11a 8CPU            2.6 :   1200 (+154,=857,-189),  48.5 %

   vs.                             :  games (   +,   =,   -),   (%)
   CFish 050920 + SV-1739          :    200 (  11, 158,  31),  45.0
   Lc0 26.1 + J92-100 (c11)        :    200 (  26, 137,  37),  47.3
   SF 170820 + SV-1705             :    200 (  26, 142,  32),  48.5
   Allie 120820 + LS 15.0          :    200 (  34, 134,  32),  50.5
   Lc0 26.2 + SV-5300              :    200 (  31, 140,  29),  50.5
   SF 170820 + SV-2257             :    200 (  26, 146,  28),  49.5

7) SF 170820 + SV-2257       0.0 :   1200 (+128,=899,-173),  48.1 %

   vs.                             :  games (   +,   =,   -),   (%)
   CFish 050920 + SV-1739          :    200 (   8, 157,  35),  43.3
   Lc0 26.1 + J92-100 (c11)        :    200 (  24, 142,  34),  47.5
   SF 170820 + SV-1705             :    200 (  18, 164,  18),  50.0
   Allie 120820 + LS 15.0          :    200 (  26, 138,  36),  47.5
   Lc0 26.2 + SV-5300              :    200 (  24, 152,  24),  50.0
   CFish 11a 8CPU                  :    200 (  28, 146,  26),  50.5
Top

Return to “Computer Chess Club: Tournaments and Matches”