Stockfish NNUE SV Tests

Dann Corbit · Post by **Dann Corbit** » Wed Aug 05, 2020 12:58 am

I tried combining your files to make a comprehensive list but it is not clear to me what Stockfish 2108 is (normal or NNUE) and some other things are hard to understand.

Code: Select all

    Program                          Elo    +   -   Games   Score   Av.Op.  Draws

  1 Stockfish 2108                 : 3286   23  22   400    67.6 %   3158   53.8 %
  2 Stockfish NNUE SV 0252         : 3269   22  21   400    65.5 %   3158   58.5 %
  3 Stockfish NNUE SF 1209         : 3265   25  24   300    65.0 %   3158   58.7 %
  4 Stockfish qh-fine-0803-5       : 3263   21  20   400    64.8 %   3158   60.5 %
  5 Stockfish NNUE SV  1515        : 3263   30  28   200    64.8 %   3158   61.5 %
  6 Stockfish SV 1442              : 3263   16  16   660    64.7 %   3158   60.3 %
  7 Stockfish NNUE SV 1817         : 3262   18  18   608    64.6 %   3158   56.1 %
  8 Stockfish NNUE SV 1515         : 3260   15  15   801    64.4 %   3158   59.1 %
  9 Stockfish NNUE SV 111          : 3254   25  25   300    63.5 %   3158   57.7 %
 10 Stockfish NNUE SV 2108         : 3250   23  22   350    63.0 %   3158   60.9 %
 11 Stockfish qh-fine-0803-5       : 3245   25  23   200    62.3 %   3158   72.5 %
 12 Stockfish NNUE SV 2257         : 3243   21  20   340    62.1 %   3158   65.9 %
 13 Stockfish NNUE SV 2151         : 3243   15  15   800    62.0 %   3158   61.0 %
 14 Stockfish NNUE SV 2214         : 3242   21  20   362    61.9 %   3158   65.2 %
 15 Stockfish NNUE SV 1743         : 3232   36  32   100    60.5 %   3158   73.0 %
 16 Stockfish NNUE SV 0511         : 3228   16  16   590    60.0 %   3158   66.8 %
 17 Stockfish NNUE 19072019 bmi2   : 3227   14  13   960    59.8 %   3158   61.4 %
 18 Stockfish NNUE SV 1130         : 3216   12  12  1025    58.2 %   3158   67.7 %
 19 Stockfish SV 220720            : 3210   21  21   360    57.5 %   3158   65.6 %
 20 Stockfish NNUE SV 2344         : 3207   28  26   150    57.0 %   3158   75.3 %
 21 Stockfish NNUE SV 210720       : 3198   15  15   700    55.8 %   3158   65.9 %
 22 Stockfish 170720 x64 bmi2      : 3158    4   4 10006    38.2 %   3242   62.4 %


    Program                          Elo    +   -   Games   Score   Av.Op.  Draws

  1 Stockfish SV 200720            : 3213   19  19   360    53.5 %   3188   71.4 %
  2 Stockfish NNUE SV 200720       : 3211   21  20   360    53.2 %   3188   67.5 %
  3 Stockfish 110720 x64 bmi2      : 3188   14  14   720    46.7 %   3212   69.4 %

Dann Corbit · Post by **Dann Corbit** » Wed Aug 05, 2020 1:09 am

As a guess, maybe this is what was intended:

Code: Select all

    Program                          Elo    +   -   Games   Score   Av.Op.  Draws

  1 Stockfish NNUE SV 0252         : 3269   22  21   400    65.5 %   3158   58.5 %
  2 Stockfish NNUE SV 2108         : 3269   16  16   750    65.5 %   3158   57.1 %
  3 Stockfish NNUE SV 1209         : 3265   25  24   300    65.0 %   3158   58.7 %
  4 Stockfish NNUE SV 1442         : 3263   16  16   660    64.7 %   3158   60.3 %
  5 Stockfish NNUE SV 1817         : 3262   18  18   608    64.6 %   3158   56.1 %
  6 Stockfish NNUE SV 1515         : 3261   13  13  1001    64.4 %   3158   59.5 %
  7 Stockfish qh-fine-0803-5       : 3257   16  16   600    63.9 %   3158   64.5 %
  8 Stockfish NNUE SV 111          : 3254   25  25   300    63.5 %   3158   57.7 %
  9 Stockfish NNUE SV 2257         : 3243   21  20   340    62.1 %   3158   65.9 %
 10 Stockfish NNUE SV 2151         : 3243   15  15   800    62.0 %   3158   61.0 %
 11 Stockfish NNUE SV 2214         : 3242   21  20   362    61.9 %   3158   65.2 %
 12 Stockfish NNUE SV 1743         : 3232   36  32   100    60.5 %   3158   73.0 %
 13 Stockfish NNUE SV 0511         : 3228   16  16   590    60.0 %   3158   66.8 %
 14 Stockfish NNUE 19072019 bmi2   : 3227   14  13   960    59.8 %   3158   61.4 %
 15 Stockfish NNUE SV 1130         : 3216   12  12  1025    58.2 %   3158   67.7 %
 16 Stockfish NNUE SV 220720       : 3210   21  21   360    57.5 %   3158   65.6 %
 17 Stockfish NNUE SV 2344         : 3207   28  26   150    57.0 %   3158   75.3 %
 18 Stockfish NNUE SV 210720       : 3198   15  15   700    55.8 %   3158   65.9 %
 19 Stockfish 170720 x64 bmi2      : 3158    4   4 10006    38.2 %   3242   62.4 %


    Program                          Elo    +   -   Games   Score   Av.Op.  Draws

  1 Stockfish NNUE SV 200720       : 3212   14  14   720    53.3 %   3188   69.4 %
  2 Stockfish 110720 x64 bmi2      : 3188   14  14   720    46.7 %   3212   69.4 %

mehmet123 · Post by **mehmet123** » Wed Aug 05, 2020 4:58 am

Dann Corbit wrote: ↑Wed Aug 05, 2020 12:58 am I tried combining your files to make a comprehensive list but it is not clear to me what Stockfish 2108 is (normal or NNUE) and some other things are hard to understand.

Thanks for list. I think there is nothing complex to understand. I only forgot to write NNUE having a match against Stockfish 170720. Because of that in pgn files only write Stockfish 2108. But I corrected this in my post.

mehmet123 wrote: ↑Mon Aug 03, 2020 9:46 pm Test of 20200803-2108 Net

Program Elo + - Games Score Av.Op. Draws

1 Stockfish NNUE SV 2108 : 2464 23 22 400 67.6 % 2336 53.8 %
2 Stockfish 170720 x64 bmi2 : 2336 22 23 400 32.4 % 2464 53.8 %

Individual statistics:

1 Stockfish NNUE SV 2108 : 2464 400 (+163,=215,- 22), 67.6 %

Stockfish 170720 x64 bmi2 : 400 (+163,=215,- 22), 67.6 %

2 Stockfish 170720 x64 bmi2 : 2336 400 (+ 22,=215,-163), 32.4 %

Stockfish NNUE SV 2108 : 400 (+ 22,=215,-163), 32.4 %

Game Conditions:Cutechess Gui, 1 Core (i7 9750h), 30 sec + 0.5 sec TC, No opening book, Ponder off, 64 Mb Hash
Compilation: Stockfish NNUE 250720 x64 Haswell BMI2 256 mingw10 (From "ChessMan"/The Outskirts Chess Forum Member)
http://www.mediafire.com/file/pja5fhgii ... 0.pgn/file

Great record ( +128 elo) at this time condition (30 sec + 0.5 sec )

mehmet123 · Post by **mehmet123** » Wed Aug 05, 2020 7:57 pm

Test of 20200804-1817 Net

Program Elo + - Games Score Av.Op. Draws

1 Stockfish NNUE SV 1817 : 2445 30 26 150 62.7 % 2355 72.0 %
2 Stockfish 170720 x64 bmi2 : 2355 26 30 150 37.3 % 2445 72.0 %

Individual statistics:

1 Stockfish NNUE SV 1817 : 2445 150 (+ 40,=108,- 2), 62.7 %

Stockfish 170720 x64 bmi2 : 150 (+ 40,=108,- 2), 62.7 %

2 Stockfish 170720 x64 bmi2 : 2355 150 (+ 2,=108,- 40), 37.3 %

Stockfish NNUE SV 1817 : 150 (+ 2,=108,- 40), 37.3 %

Arena Gui, 6 Core (i7 9750h), 2 min + 1 sec TC, Balsa 5 Move Opening Book, 512 Mb Hash, Ponder Off
Compilation: Stockfish NNUE 250720 x64 Haswell BMI2 256 mingw10 (From "ChessMan"/The Outskirts Chess Forum Member)
http://www.mediafire.com/file/za0n2p6oc ... 5.pgn/file

Amazing record (+90 elo) at these conditions (6 core, 2 min + 1 sec). Previous record (+74 elo) belongs to Stockfish NNUE SV 1743.

The performance of Stockfish NNUE SV 1817 at 1 core, 1 min + 0.5 sec is +104 elo (x node/move)
The performance of Stockfish NNUE SV 1817 at 6 core, 2 min + 1 sec is +90 elo. (10x node/move)
According to this test, there is no significant scaling problem of this chess engine.

Jouni · Post by **Jouni** » Wed Aug 05, 2020 10:20 pm

No scaling problem in SF framework either:
ELO: 92.77 at 10+0.1 th 1
ELO: 89.01 at 20+0.2 th 8

mehmet123 · Post by **mehmet123** » Wed Aug 05, 2020 11:54 pm

Test of 20200805-1512 Net

Program Elo + - Games Score Av.Op. Draws

1 Stockfish NNUE SV 1512 : 2452 19 18 560 64.6 % 23.48 57.3 %
2 Stockfish 170720 x64 bmi2 :2348 18 19 560 35.4 % 2452 57.3 %

Individual statistics:

1 Stockfish NNUE SV 1512 : 2452 560 (+201,=321,- 38), 64.6 %

Stockfish 170720 x64 bmi2 : 560 (+201,=321,- 38), 64.6 %

2 Stockfish 170720 x64 bmi2 : 2348 560 (+ 38,=321,-201), 35.4 %

Stockfish NNUE SV 1512 : 560 (+ 38,=321,-201), 35.4 %

Game Conditions: Cutechess Gui, 1 Core (i7 9750h), 1 min + 0.5 sec TC, Balsa 5 Move Opening Book, 128 Mb Hash, Ponder Off
Compilation: Stockfish NNUE 250720 x64 Haswell BMI2 256 mingw10
http://www.mediafire.com/file/lro3jnrpd ... 0.pgn/file

The performance of the last 3 nets tested during this tc (1 min + 0.5 sec) is very close (+106 elo /SV 0109 , +104 elo /SV 1817 ,+104 elo /SV 1512)

mehmet123 · Post by **mehmet123** » Thu Aug 06, 2020 6:08 pm

Test of SV 20200805-1512 Net

Program Elo + - Games Score Av.Op. Draws

1 Stockfish NNUE SV 1512 : 2449 20 19 380 63.7 % 2351 65.8 %
2 Stockfish 170720 x64 bmi2 : 2351 19 20 380 36.3 % 2449 65.8 %

Individual statistics:

1 Stockfish NNUE SV 1512 : 2449 380 (+117,=250,- 13), 63.7 %

Stockfish 170720 x64 bmi2 : 380 (+117,=250,- 13), 63.7 %

2 Stockfish 170720 x64 bmi2 : 2351 380 (+ 13,=250,-117), 36.3 %

Stockfish NNUE SV 1512 : 380 (+ 13,=250,-117), 36.3 %

Game conditions: Cutechess Gui, 1 Core (i7 9750h), 2 min + 0.5 sec TC, Balsa 5 Move Opening Book, 256 Mb Hash, Ponder Off
Compilation: Stockfish NNUE 250720 x64 Haswell BMI2 256 mingw10
http://www.mediafire.com/file/0b3uhk307 ... 2.pgn/file

It's a new record ( +98 elo ) at this time control (2 min + 0.5 sec). Previous record ( +92 elo) belonged to Stockfish NNUE SV 1515.

mehmet123 · Post by **mehmet123** » Fri Aug 07, 2020 5:48 am

Test of 20200806-1802 Net

Program Elo + - Games Score Av.Op. Draws

1 Stockfish NNUE 1802 : 2446 18 18 500 63.0 % 2354 63.2 %
2 Stockfish 310720 x64 bmi2 : 2354 18 18 500 37.0 % 2446 63.2 %

Individual statistics:

1 Stockfish NNUE 1802 : 2446 500 (+157,=316,- 27), 63.0 %

Stockfish 310720 x64 bmi2 : 500 (+157,=316,- 27), 63.0 %

2 Stockfish 310720 x64 bmi2 : 2354 500 (+ 27,=316,-157), 37.0 %

Stockfish NNUE 1802 : 500 (+ 27,=316,-157), 37.0 %

Game conditions: Cutechess Gui, 1 Core (i7 9750h), 2 min + 0.5 sec TC, Balsa 5 Move Opening Book, 256 Mb Hash, Ponder Off
Compilation: Stockfish NNUE 250720 x64 Haswell BMI2 256 mingw10
http://www.mediafire.com/file/dunio6srg ... 6.pgn/file

Good performance (+92 elo) but it couldn't break the record (+98 elo) at this time control (2 min + 0.5 sec) in my test.

mehmet123 · Post by **mehmet123** » Sat Aug 08, 2020 6:02 am

Test of SV 20200807-1950 Net:

Program Elo + - Games Score Av.Op. Draws

1 Stockfish NNUE SV 1950 : 2450 17 16 560 64.0 % 2350 64.1 %
2 Stockfish 310720 x64 bmi2 : 2350 16 17 560 36.0 % 2450 64.1 %

Individual statistics:

1 Stockfish NNUE SV 1950 : 2450 560 (+179,=359,- 22), 64.0 %

Stockfish 310720 x64 bmi2 : 560 (+179,=359,- 22), 64.0 %

2 Stockfish 310720 x64 bmi2 : 2350 560 (+ 22,=359,-179), 36.0 %

Stockfish NNUE SV 1950 : 560 (+ 22,=359,-179), 36.0 %

Game conditions: Cutechess Gui, 1 Core (i7 9750h), 2 min + 0.5 sec TC, Balsa 5 Move Opening Book, 256 Mb Hash, Ponder Off
Compilation: Stockfish NNUE 250720 x64 Haswell BMI2 256 mingw10
http://www.mediafire.com/file/u1drtjn7y ... 0.pgn/file

Great record (+100 elo) at this time control (2 min + 0.5 sec). Previous record (+98 elo) belonged to Stockfish NNUE SV 1512.

mehmet123 · Post by **mehmet123** » Sat Aug 08, 2020 12:06 pm

Test of 20200808-0817 Net

Program Elo + - Games Score Av.Op. Draws

1 Stockfish NNUE SV 0817 : 2453 20 19 400 64.9 % 2347 63.2 %
2 Stockfish 310720 x64 bmi2 : 2347 19 20 400 35.1 % 2453 63.2 %

Individual statistics:

1 Stockfish NNUE SV 0817 : 2453 400 (+133,=253,- 14), 64.9 %

Stockfish 310720 x64 bmi2 : 400 (+133,=253,- 14), 64.9 %

2 Stockfish 310720 x64 bmi2 : 2347 400 (+ 14,=253,-133), 35.1 %

Stockfish NNUE SV 0817 : 400 (+ 14,=253,-133), 35.1 %

Game Conditions: Cutechess Gui, 1 Core (i7 9750h), 1 min + 0.5 sec TC, Balsa 5 Move Opening Book, 128 Mb Hash, Ponder Off
Compilation: Stockfish NNUE 250720 x64 Haswell BMI2 256 mingw10
http://www.mediafire.com/file/9z673of4l ... 1.pgn/file

The performance of Stockfish NNUE is in the range of +104 elo and +106 elo in my last 4 tests at this time control (1 min + 0.5 sec).

Stockfish NNUE SV Tests

Re: Stockfish NNUE SV Tests

Re: Stockfish NNUE SV Tests

Re: Stockfish NNUE SV Tests

Re: Stockfish NNUE SV Tests

Re: Stockfish NNUE SV Tests

Re: Stockfish NNUE SV Tests

Re: Stockfish NNUE SV Tests

Re: Stockfish NNUE SV Tests

Re: Stockfish NNUE SV Tests

Re: Stockfish NNUE SV Tests