A quick LC0 experiment. Which NN from my drawer is best?

[Dann Corbit]

I ran a tournament using various Nets that I was told were good ones at one time or another.
Here is the result:

Code: Select all

# Program               Elo    +   -   Games   Score   Av.Op.  Draws 
- --------------------  ----   --  --   ---    ------   ----   ------
1 Lc0-68002           : 3430   15  15   500    55.2 %   3394   74.8 %
2 Lc0-68501           : 3415   14  14   500    52.6 %   3397   78.0 %
3 Lc0-j98.1-16        : 3414   14  14   500    52.4 %   3397   78.0 %
4 Lc0-384x30-t60-4619 : 3399   15  15   500    49.8 %   3400   76.8 %
5 Lc0-384x30-t60-3010 : 3395   15  15   500    49.1 %   3401   75.4 %
6 Lc0-384x30-t40-2036 : 3347   16  16   500    40.9 %   3411   71.4 %

The games:

[Milos]

I ran a tournament using various Nets that I was told were good ones at one time or another.
Here is the result:

Code: Select all

# Program               Elo    +   -   Games   Score   Av.Op.  Draws 
- --------------------  ----   --  --   ---    ------   ----   ------
1 Lc0-68002           : 3430   15  15   500    55.2 %   3394   74.8 %
2 Lc0-68501           : 3415   14  14   500    52.6 %   3397   78.0 %
3 Lc0-j98.1-16        : 3414   14  14   500    52.4 %   3397   78.0 %
4 Lc0-384x30-t60-4619 : 3399   15  15   500    49.8 %   3400   76.8 %
5 Lc0-384x30-t60-3010 : 3395   15  15   500    49.1 %   3401   75.4 %
6 Lc0-384x30-t40-2036 : 3347   16  16   500    40.9 %   3411   71.4 %

The games:

This certainly tells us when different nets play against each other which one is the strongest. However, that is I believe of a little interest. What ppl would like to know which one would perform the best against the latest SF or a pool of AB engines.

[Dann Corbit]

I ran a tournament using various Nets that I was told were good ones at one time or another.
Here is the result:

Code: Select all

# Program               Elo    +   -   Games   Score   Av.Op.  Draws 
- --------------------  ----   --  --   ---    ------   ----   ------
1 Lc0-68002           : 3430   15  15   500    55.2 %   3394   74.8 %
2 Lc0-68501           : 3415   14  14   500    52.6 %   3397   78.0 %
3 Lc0-j98.1-16        : 3414   14  14   500    52.4 %   3397   78.0 %
4 Lc0-384x30-t60-4619 : 3399   15  15   500    49.8 %   3400   76.8 %
5 Lc0-384x30-t60-3010 : 3395   15  15   500    49.1 %   3401   75.4 %
6 Lc0-384x30-t40-2036 : 3347   16  16   500    40.9 %   3411   71.4 %

The games:

This certainly tells us when different nets play against each other which one is the strongest. However, that is I believe of a little interest. What ppl would like to know which one would perform the best against the latest SF or a pool of AB engines.

You are absolutely right in that an engine test only measured exactly what was measured in the experiment and only under the same conditions as the experiment. E.g. if you have different GPUs, the results could be different. If you use different time control, the results could be different. And, as you mention, it might perform differently against different competition.

But it was enough for me to prefer 68002 until I do some other test that tells me otherwise, or someone else runs a really good test that teaches everyone which net is best for more general conditions.