LC0 11248 gauntlet vs new LC0 generation

[majortom]

LCZero 11248 (714 nodes per move) vs LCZero 32367 (714 nodes per move)

Code: Select all

     Program               Elo      +      -   Games   Score   Av.Op.  Draws

   1 LC0 11248 N714    :    0.0    0.0    0.0   200    54.8 %    -34   51.5 %
   2 LC0 32367 N714    :  -33.6   35.0   35.0   200    45.3 %      0   51.5 %

Code: Select all

1 LC0 11248 N714 :  0.0 200 (+ 58,=103,- 39), 54.8 %

LC0 32367 N714   : 200 (+ 58,=103,- 39), 54.8 %

2 LC0 32367 N714 :-33.6 200 (+ 39,=103,- 58), 45.3 %

LC0 11248 N714   : 200 (+ 39,=103,- 58), 45.3 %

Games

[majortom]

Code: Select all

     Program               Elo      +      -   Games   Score   Av.Op.  Draws

   1 LC0 11248 N714    :    0.0    0.0    0.0  1400    62.5 %    -91   45.1 %
   2 LC0 32367 N714    :  -33.6   32.5   32.5   200    45.3 %      0   51.5 %
   3 LC0 32177 N714    :  -42.5   34.0   34.0   200    44.0 %      0   46.0 %
   4 LC0 32015 N714    :  -82.6   37.3   37.3   200    38.5 %      0   45.0 %
   5 LC0 31741 N714    :  -88.2   37.4   37.4   200    37.8 %      0   45.5 %
   6 LC0 31885 N714    : -113.0   35.7   35.7   200    34.5 %      0   47.0 %
   7 LC0 31687 N714    : -136.9   36.9   36.9   200    31.5 %      0   42.0 %
   8 LC0 31633 N714    : -143.1   36.4   36.4   200    30.8 %      0   38.5 %

[majortom]

Number of nodes is increased in 10 times.

LCZero 11248 (7140 nodes per move) vs LCZero 32458(7140 nodes per move)

Code: Select all

    Program                Elo      +      -   Games   Score   Av.Op.  Draws

   1 LC0 32458 N7140    :   77.0   81.2   81.2    38    60.5 %      0   63.2 %
   2 LC0 11248 N7140    :    0.0    0.0    0.0    38    39.5 %     77   63.2 %

Code: Select all

1 LC0 32458 N7140 : 77.0 38 (+ 11,= 24,-  3), 60.5 %

LC0 11248 N7140   : 38 (+ 11,= 24,-  3), 60.5 %

2 LC0 11248 N7140 :  0.0 38 (+  3,= 24,- 11), 39.5 %

LC0 32458 N7140   : 38 (+  3,= 24,- 11), 39.5 %

Games

[majortom]

LCZero 11248 (7140 nodes per move) vs LCZero 32458(7140 nodes per move)

Code: Select all

     Program                Elo      +      -   Games   Score   Av.Op.  Draws

   1 LC0 32458 N7140    :   54.2   61.9   61.9    60    57.5 %      0   61.7 %
   2 LC0 11248 N7140    :    0.0    0.0    0.0    60    42.5 %     54   61.7 %

Code: Select all

1 LC0 32458 N7140 : 54.2 60 (+ 16,= 37,-  7), 57.5 %

LC0 11248 N7140   : 60 (+ 16,= 37,-  7), 57.5 %

2 LC0 11248 N7140 :  0.0 60 (+  7,= 37,- 16), 42.5 %

LC0 32458 N7140   : 60 (+  7,= 37,- 16), 42.5 %

Games

[brianr]

FWIW:

Code: Select all

Score of ID-32585 vs ID-11248: 61 - 63 - 176  [0.497] 300
Elo difference: -2.32 +/- 25.28

At 320 games it was exactly even.

Match with v20.1 net 32585 v 11248 at 0:10+0.4 with 6 piece TBs and 2 move book on a GTX-1070.

Of course, 32585 is not 32458.

[brianr]

FWIW:

Code: Select all

Score of ID-32585 vs ID-11248: 61 - 63 - 176  [0.497] 300
Elo difference: -2.32 +/- 25.28

At 320 games it was exactly even.

Match with v20.1 net 32585 v 11248 at 0:10+0.4 with 6 piece TBs and 2 move book on a GTX-1070.

Of course, 32585 is not 32458.

Update/correction. I ran a couple more matches. I changed the "new" net to 32747, which I think is the one selected for TCEC. The first was at the same time controls as the earlier one.

Code: Select all

Score of ID-32742 vs ID-11248: 19 - 25 - 56  [0.470] 100
Elo difference: -20.87 +/- 45.34

This result was in line with the earlier match with the other 30 series net.

However, there was talk on the Leela Discord chat about the newer nets scaling better than the older ones, so I did another match with tc=1:00+2.0 (1 min plus 2 seconds per move). The results were somewhat startling.

Code: Select all

Score of ID-32742 vs ID-11248: 27 - 12 - 61  [0.575] 100
Elo difference: 52.51 +/- 42.39

I suspect the difference will be even greater on stronger hardware.
Also, the 11248 net was trained with the 50 move bug.

[majortom]

LCZero 11248 (7140 nodes per move) vs LCZero 32458(7140 nodes per move)

Code: Select all

     Program                Elo      +      -   Games   Score   Av.Op.  Draws

   1 LC0 32458 N7140    :   66.0   52.1   52.1   100    59.0 %      0   62.0 %
   2 LC0 11248 N7140    :    0.0    0.0    0.0   100    41.0 %     66   62.0 %

Code: Select all

1 LC0 32458 N7140 : 66.0 100 (+ 28,= 62,- 10), 59.0 %

LC0 11248 N7140   : 100 (+ 28,= 62,- 10), 59.0 %

2 LC0 11248 N7140 :  0.0 100 (+ 10,= 62,- 28), 41.0 %

LC0 32458 N7140   : 100 (+ 10,= 62,- 28), 41.0 %

Games