Tournament SF+NNUE, Lc0, Ethereal, SF+Classical Eval, Komodo, Stockfish+NNUE PO, (15m+15s) (8 Rounds)

[mwyoung]

Tournament SF+NNUE, Lc0, Ethereal, SF+Classical Eval, Komodo, Stockfish+NNUE PO, (15m+15s) (8 Rounds)

Hardware 2950x, RTX 2080 Ti

Stockfish 260820 (NN-1705)
Lc0 26.1 (NN-J92-70)
Stockfish+NNUE PO (NN-1705)
Ethereal 12.25
Stockfish 240820+Classical Eval Only
Komodo 14

Ponder off.
TC=15m+15s
32 threads.
4 Gb hash.
6 man TB, and 10, 7 man TB.
Opening book 6 moves.
Default settings.

Live Stream:

[mwyoung]

Update 1

Code: Select all

Result:
----------------------------------------------------------------------------------------
  #  name                        games    wins   draws  losses   score    los%  elo+/-
  1. Stockfish 260820               11       2       9       0     6.5    92.1    63.9
  2. Lc0 v0.26.1                     7       0       7       0     3.5-nan(ind)    -0.0
  3. SF+NNUE PO 290720 x64 avx2      7       0       7       0     3.5-nan(ind)    -0.0
  4. Stockfish 240820                6       0       6       0     3.0-nan(ind)    -0.0
  5. Komodo 14 64-bit                7       0       6       1     3.0    15.9   -50.0
  6. Ethereal 12.25 (POPCNT)         6       0       5       1     2.5    15.9   -58.5

Cross table:
----------------------------------------------------------------------------------------
  #  name                           score   games         1         2         3         4         5         6
  1. Stockfish 260820                 6.5      11         x        ==        ==        ==      1===         1
  2. Lc0 v0.26.1                      3.5       7        ==         x                   =         =       ===
  3. SF+NNUE PO 290720 x64 avx2       3.5       7        ==                   x        ==        ==         =
  4. Stockfish 240820                 3.0       6        ==         =        ==         x                   =
  5. Komodo 14 64-bit                 3.0       7      0===         =        ==                   x          
  6. Ethereal 12.25 (POPCNT)          2.5       6         0       ===         =         =                   x

Tech:
----------------------------------------------------------------------------------------

Tech (average nodes, depths, time/m per move, others per game), counted for computing moves only, ignored moves with zero nodes:
  #  name                          nodes/m         NPS  depth/m   time/m    moves     time
  1. Stockfish 260820              720695K    24373231     49.5     29.6     52.5   1553.7
  2. Lc0 v0.26.1                      905K       30098     14.0     30.1     52.0   1563.0
  3. SF+NNUE PO 290720 x64 avx2    562655K    22139803     51.4     25.4     74.6   1895.1
  4. Stockfish 240820             1179573K    40773385     56.9     28.9     59.0   1706.9
  5. Komodo 14 64-bit             1095230K    38714735     45.3     28.3     63.7   1802.5
  6. Ethereal 12.25 (POPCNT)      1193337K    39848765     38.0     29.9     54.8   1642.1
     all ---                       756712K    27148695     43.7     28.5     58.9   1682.0

[mwyoung]

Update 2.

Code: Select all

Result:
----------------------------------------------------------------------------------------
  #  name                        games    wins   draws  losses   score    los%  elo+/-
  1. SF+NNUE PO 290720 x64 avx2     22       3      19       0    12.5    95.8    47.7
  2. Stockfish 260820               19       3      16       0    11.0    95.8    55.3
  3. Lc0 v0.26.1                    18       1      17       0     9.5    84.1    19.3
  4. Stockfish 240820               17       0      16       1     8.0    15.9   -20.5
  5. Komodo 14 64-bit               20       0      17       3     8.5     4.2   -52.5
  6. Ethereal 12.25 (POPCNT)        14       0      11       3     5.5     4.2   -75.6

Cross table:
----------------------------------------------------------------------------------------
  #  name                           score   games         1         2         3         4         5         6
  1. SF+NNUE PO 290720 x64 avx2      12.5      22         x      ====      ====      ====    ===11=      =1==
  2. Stockfish 260820                11.0      19      ====         x      ====       ===     1====       1=1
  3. Lc0 v0.26.1                      9.5      18      ====      ====         x      ==1=       ===       ===
  4. Stockfish 240820                 8.0      17      ====       ===      ==0=         x      ====        ==
  5. Komodo 14 64-bit                 8.5      20    ===00=     0====       ===      ====         x        ==
  6. Ethereal 12.25 (POPCNT)          5.5      14      =0==       0=0       ===        ==        ==         x

Tech:
----------------------------------------------------------------------------------------

Tech (average nodes, depths, time/m per move, others per game), counted for computing moves only, ignored moves with zero nodes:
  #  name                          nodes/m         NPS  depth/m   time/m    moves     time
  1. SF+NNUE PO 290720 x64 avx2    589832K    22938312     52.6     25.7     72.3   1859.6
  2. Stockfish 260820              670448K    23654784     49.6     28.3     58.0   1643.9
  3. Lc0 v0.26.1                      827K       29014     12.5     28.5     60.6   1726.8
  4. Stockfish 240820             1072077K    40667080     53.0     26.4     72.1   1899.6
  5. Komodo 14 64-bit             1049471K    38210085     43.8     27.5     65.4   1796.3
  6. Ethereal 12.25 (POPCNT)      1195321K    40922769     37.1     29.2     55.9   1631.5
     all ---                       729037K    27273384     42.7     27.4     64.5   1766.3

[mwyoung]

Update 3

Code: Select all

Result:
----------------------------------------------------------------------------------------
  #  name                        games    wins   draws  losses   score    los%  elo+/-
  1. SF+NNUE PO 290720 x64 avx2     31       3      28       0    17.0    95.8    33.7
  2. Stockfish 260820               28       3      25       0    15.5    95.8    37.4
  3. Lc0 v0.26.1                    28       2      26       0    15.0    92.1    24.9
  4. Stockfish 240820               24       0      23       1    11.5    15.9   -14.5
  5. Ethereal 12.25 (POPCNT)        23       0      20       3    10.0     4.2   -45.6
  6. Komodo 14 64-bit               30       0      26       4    13.0     2.3   -46.6

Cross table:
----------------------------------------------------------------------------------------
  #  name                           score   games         1         2         3         4         5         6
  1. SF+NNUE PO 290720 x64 avx2      17.0      31         x   =======   =======     =====     =1===   ===11==
  2. Stockfish 260820                15.5      28   =======         x      ====     =====     1=1==   1======
  3. Lc0 v0.26.1                     15.0      28   =======      ====         x     ==1==    ======    =====1
  4. Stockfish 240820                11.5      24     =====     =====     ==0==         x       ===    ======
  5. Ethereal 12.25 (POPCNT)         10.0      23     =0===     0=0==    ======       ===         x      ====
  6. Komodo 14 64-bit                13.0      30   ===00==   0======    =====0    ======      ====         x

Tech:
----------------------------------------------------------------------------------------

Tech (average nodes, depths, time/m per move, others per game), counted for computing moves only, ignored moves with zero nodes:
  #  name                          nodes/m         NPS  depth/m   time/m    moves     time
  1. SF+NNUE PO 290720 x64 avx2    620953K    22951696     52.9     27.1     63.8   1727.1
  2. Stockfish 260820              671727K    23144990     49.2     29.0     55.0   1595.2
  3. Lc0 v0.26.1                      749K       26857     12.3     27.9     64.2   1789.1
  4. Stockfish 240820             1082531K    40394824     53.5     26.8     68.0   1823.4
  5. Ethereal 12.25 (POPCNT)      1131489K    40433733     37.0     28.0     60.6   1694.8
  6. Komodo 14 64-bit             1073315K    38361685     44.8     28.0     62.8   1757.1
     all ---                       728726K    26889253     41.7     27.8     62.3   1730.2

[Alayan]

Looks like with this drawish opening book, even Ethereal is really hard to beat.

And less than 100 elo performance gap from top to bottom in this run so far. Small sample size, but it still tells us something.

[mwyoung]

Looks like with this drawish opening book, even Ethereal is really hard to beat.

And less than 100 elo performance gap from top to bottom in this run so far. Small sample size, but it still tells us something.

What did this same drawish book tell you at 3m+2s?

[mwyoung]

Looks like with this drawish opening book, even Ethereal is really hard to beat.

And less than 100 elo performance gap from top to bottom in this run so far. Small sample size, but it still tells us something.

What did this same drawish book tell you at 3m+2s?

I will show you, same "drawish opening book", same settings, only the time control has changed. 3m+2s vs here 15m+15s.

Code: Select all

Result:
----------------------------------------------------------------------------------------
  #  name                        games    wins   draws  losses   score    los%  elo+/-
  1. SF+NNUE PO 290720 x64 avx2     60      18      42       0    39.0   100.0   107.5
  2. Stockfish 260820               60      10      50       0    35.0    99.9    58.5
  3. Stockfish 240820               60       8      42      10    29.0    31.9   -11.6
  4. Lc0 v0.26.1                    60       6      51       3    31.5    84.1    17.4
  5. Komodo 14 64-bit               60       4      42      14    25.0     0.9   -58.5
  6. Ethereal 12.25 (POPCNT)        60       0      41      19    20.5     0.0  -113.9

[MMarco]

Looks like with this drawish opening book, even Ethereal is really hard to beat.

And less than 100 elo performance gap from top to bottom in this run so far. Small sample size, but it still tells us something.

What did this same drawish book tell you at 3m+2s?

I will show you, same "drawish opening book", same settings, only the time control has changed. 3m+2s vs here 15m+15s.

Code: Select all

Result:
----------------------------------------------------------------------------------------
  #  name                        games    wins   draws  losses   score    los%  elo+/-
  1. SF+NNUE PO 290720 x64 avx2     60      18      42       0    39.0   100.0   107.5
  2. Stockfish 260820               60      10      50       0    35.0    99.9    58.5
  3. Stockfish 240820               60       8      42      10    29.0    31.9   -11.6
  4. Lc0 v0.26.1                    60       6      51       3    31.5    84.1    17.4
  5. Komodo 14 64-bit               60       4      42      14    25.0     0.9   -58.5
  6. Ethereal 12.25 (POPCNT)        60       0      41      19    20.5     0.0  -113.9

Does SF NNUE "PO" stand for Ponder On? On 32 threads?

[mwyoung]

Looks like with this drawish opening book, even Ethereal is really hard to beat.

And less than 100 elo performance gap from top to bottom in this run so far. Small sample size, but it still tells us something.

What did this same drawish book tell you at 3m+2s?

I will show you, same "drawish opening book", same settings, only the time control has changed. 3m+2s vs here 15m+15s.

Code: Select all

Result:
----------------------------------------------------------------------------------------
  #  name                        games    wins   draws  losses   score    los%  elo+/-
  1. SF+NNUE PO 290720 x64 avx2     60      18      42       0    39.0   100.0   107.5
  2. Stockfish 260820               60      10      50       0    35.0    99.9    58.5
  3. Stockfish 240820               60       8      42      10    29.0    31.9   -11.6
  4. Lc0 v0.26.1                    60       6      51       3    31.5    84.1    17.4
  5. Komodo 14 64-bit               60       4      42      14    25.0     0.9   -58.5
  6. Ethereal 12.25 (POPCNT)        60       0      41      19    20.5     0.0  -113.9

Does SF NNUE "PO" stand for Ponder On? On 32 threads?

No.

sf-nnue po 290720
@FireFather FireFather released this on Jul 28

stockfish+nnue po (play only)

The download includes fast optimized PGO bmi2, avx2, & popc (sse42) binaries

https://github.com/FireFather/sf-nnue/releases

[mwyoung]

Update 4.

Code: Select all

Result:
----------------------------------------------------------------------------------------
  #  name                        games    wins   draws  losses   score    los%  elo+/-
  1. Stockfish 260820               36       4      32       0    20.0    97.7    38.8
  2. SF+NNUE PO 290720 x64 avx2     35       3      32       0    19.0    95.8    29.9
  3. Lc0 v0.26.1                    34       2      32       0    18.0    92.1    20.5
  4. Stockfish 240820               35       1      32       2    17.0    28.2    -9.9
  5. Komodo 14 64-bit               37       1      32       4    17.0     9.0   -28.2
  6. Ethereal 12.25 (POPCNT)        33       0      28       5    14.0     1.3   -53.1

Cross table:
----------------------------------------------------------------------------------------
  #  name                           score   games         1         2         3         4         5         6
  1. Stockfish 260820                20.0      36         x  ========    ======  =======1  1=======    1=1===
  2. SF+NNUE PO 290720 x64 avx2      19.0      35  ========         x  ========    ======   ===11==    =1====
  3. Lc0 v0.26.1                     18.0      34    ======  ========         x    ==1===   =====1=   =======
  4. Stockfish 240820                17.0      35  =======0    ======    ==0===         x  ========   ======1
  5. Komodo 14 64-bit                17.0      37  0=======   ===00==   =====0=  ========         x   =====1=
  6. Ethereal 12.25 (POPCNT)         14.0      33    0=0===    =0====   =======   ======0   =====0=         x

Tech:
----------------------------------------------------------------------------------------

Tech (average nodes, depths, time/m per move, others per game), counted for computing moves only, ignored moves with zero nodes:
  #  name                          nodes/m         NPS  depth/m   time/m    moves     time
  1. Stockfish 260820              645276K    23038183     48.6     28.0     59.8   1675.9
  2. SF+NNUE PO 290720 x64 avx2    620298K    22781628     52.7     27.2     62.1   1690.5
  3. Lc0 v0.26.1                      820K       29477     12.5     27.8     63.4   1764.2
  4. Stockfish 240820             1073594K    40572786     54.4     26.5     69.5   1839.4
  5. Komodo 14 64-bit             1064306K    38341489     45.2     27.8     62.7   1741.3
  6. Ethereal 12.25 (POPCNT)      1130476K    40450544     37.0     27.9     61.3   1714.1
     all ---                       744516K    27709871     42.1     27.5     63.1   1737.4