LCZero progress

[majortom]

Started a test (round-robin tournament, 714 nodes per move) to check if there is some regression in NN training.

Code: Select all

     Rank Name                     Elo     +/-   Games   Score   Draws
   1 LC0 253_442d9a80 N714          37      55      56   55.4%   64.3%
   2 LC0 262_891f63a9 N714           0      65      56   50.0%   50.0%
   3 LC0 239_0df31cb9 N714         -37      57      56   44.6%   60.7%

N714 - nodes=714

[jp]

majortom, what GUI, etc. do you use to run these tournaments?
I see you have different #nodes, also some depth & some nodes.
I've been looking for something that lets you mix different values & different time/depth/#nodes
for the competing engines.

[majortom]

1. what GUI, etc. do you use to run these tournaments?

I use cutechess-cli only.

I see you have different #nodes, also some depth & some nodes.
I've been looking for something that lets you mix different values & different time/depth/#nodes
for the competing engines.

You can see my calculation here.
Of course, I can't be sure at 100% that my ratio is correct, but I hope I'm not far from the truth.

[majortom]

I use SF at depth=15 as a very quick and strong counter.
SF at depth=30 was added for undestanding how much stronger it is at solid strength.

[majortom]

Code: Select all

     Rank Name                     Elo     +/-   Games   Score   Draws
   1 LC0 253_442d9a80 N714          11      36     162   51.5%   53.7%
   2 LC0 239_0df31cb9 N714           6      38     162   50.9%   50.0%
   3 LC0 262_891f63a9 N714         -17      39     164   47.6%   47.6%

[majortom]

Code: Select all

     Rank Name                      Elo     +/-   Games   Score   Draws
   1 LC0 239_0df31cb9 N714           18      33     212   50.9%   49.1%
   2 LC0 253_442d9a80 N714           16      33     212   50.7%   51.4%
   3 LC0 262_891f63a9 N714            0      34     212   48.3%   47.6%

[jp]

Thanks. I take it you're working out the times it takes for LC0 173 to get a number of nodes
(and x2.8 for larger net), and then just setting time controls in cutechess-cli, right?
Do you do that for SF too --- work out times to get to depth 15 & 30?

i.e. does cutechess-cli allow different times for the two players but not directly number of nodes or depth?

Code: Select all

   # PLAYER                   :  RATING  ERROR  POINTS  PLAYED   (%)
   1 Stockfish 130418 D30     :  1064.3  128.6   193.0     200    97
   2 Stockfish 130418 D15     :   474.9   69.5   790.5    1089    73
   3 LC0 173_101eeb41 N16k    :   269.2   88.4    21.5      89    24
   4 LC0 173_101eeb41 N8k     :   181.5   82.3    32.5     200    16
   5 LC0 173_101eeb41 N4k     :   106.1   96.4    22.5     200    11
   6 LC0 173_101eeb41 N2k     :    39.5   90.2    16.0     200     8
   7 LC0 173_101eeb41 N1k     :     0.0   ----    13.0     200     7

White advantage = 55.82 +/- 13.16
Draw rate (equal opponents) = 50.00 % +/- 0.00

[majortom]

I take it you're working out the times it takes for LC0 173 to get a number of nodes
(and x2.8 for larger net)

Yes, that's why number of nodes for larger net is x2.8 times less:

Code: Select all

128x10	192x15

 1,000	   357
 2,000	   714
 4,000	 1,429
 8,000	 2,857
16,000	 5,714
32,000	11,429

then just setting time controls in cutechess-cli, right?

It's not exactly time management, engines play with fixed number of nodes.

Do you do that for SF too --- work out times to get to depth 15 & 30?

No, SF with D15 & D30 is just targets for LCZero, no matter how many playouts it uses.

i.e. does cutechess-cli allow different times for the two players but not directly number of nodes or depth?

Yes, it allows using different TC for each engine without fixed nodes or depths.

[majortom]

Code: Select all

     Rank Name                     Elo     +/-   Games   Score   Draws
   1 LC0 253_442d9a80 N714           3      29     279   50.2%   50.9%
   2 LC0 239_0df31cb9 N714           3      30     279   50.2%   45.9%
   3 LC0 262_891f63a9 N714           0      30     282   49.6%   44.7%

It seems they are equal.

[majortom]

I've interrupted the test:

Code: Select all

   # PLAYER                   :  RATING  ERROR  POINTS  PLAYED   (%)
   1 LC0 262_891f63a9 N714    :     3.8   31.8   155.0     308    50
   2 LC0 253_442d9a80 N714    :     3.1   31.4   154.5     308    50
   3 LC0 239_0df31cb9 N714    :     0.0   ----   152.5     308    50

White advantage = 37.32 +/- 11.32
Draw rate (equal opponents) = 50.00 % +/- 0.00

Games