Stockfish NNUE SV Tests

[mehmet123]

Test of SV 20200824-1705

Program Elo + - Games Score Av.Op. Draws

1 Stockfish NNUE SV 1035 : 2400 11 11 1375 50.0 % 2400 62.7 %
2 Stockfish NNUE SV 1705 : 2400 11 11 1375 50.0 % 2400 62.7 %

Individual statistics:

1 Stockfish NNUE SV 1035 : 2400 1375 (+257,=862,-256), 50.0 %

Stockfish NNUE SV 1705 : 1375 (+257,=862,-256), 50.0 %

2 Stockfish NNUE SV 1705 : 2400 1375 (+256,=862,-257), 50.0 %

Stockfish NNUE SV 1035 : 1375 (+256,=862,-257), 50.0 %


Game Conditions: Cutechess Gui, 1 Core ( i7 9750h), 100 kn/move, Balsa 5 Move Opening Book, 64 Mb Hash, Ponder Off
Compilation: Stockfish NNUE 240820 BMI2 mingw 10.2(ChessMan compile)
http://www.mediafire.com/file/ducbxf9yx ... 3.pgn/file

At 100 kn/move match Stockfish NNUE SV 1705 shows the same performance as Stockfish NNUE SV 1035.

[mehmet123]

The performance of SV 1705 against SV 1035:

-46 elo at 10 kn/move match.
-14 elo at 20 kn/move match.
same elo at 100 kn/move match.

This test shows one thing Stockfish NNUE SV 1705 scales very well unlike other SV networks. This net trained from scratch
It was unfortunate that this network was tested in the Stockfish framework in a short time control ( 10 sec + 0.1 sec).

[mehmet123]

Test of SV 20200824-1705

Program Elo + - Games Score Av.Op. Draws

1 Stockfish NNUE SV 1705 : 2464 19 18 550 67.6 % 2336 57.1 %
2 Stockfish 240820 x64 bmi2 : 2336 18 19 550 32.4 % 2464 57.1 %

Individual statistics:

1 Stockfish NNUE SV 1705 : 2464 550 (+215,=314,- 21), 67.6 %

Stockfish 240820 x64 bmi2 : 550 (+215,=314,- 21), 67.6 %

2 Stockfish 240820 x64 bmi2 : 2336 550 (+ 21,=314,-215), 32.4 %

Stockfish NNUE SV 1705 : 550 (+ 21,=314,-215), 32.4 %

Game Conditions: Cutechess Gui, 1 Core ( i7 9750h), 2 min + 0.5 sec TC, Balsa 5 Move Opening Book, 128 Mb Hash, Ponder Off
Compilation: Stockfish NNUE240820 x64 BMI2_mingw10.2 (ChessMan compile)
http://www.mediafire.com/file/dvi4t88bd ... 7.pgn/file

Amazing result. It's a record (+ 128 elo) at this time control (2 min + 0.5 sec) in my tests.
Previous record was belonged to Cfish NNUE SV 1035 (+106 elo).
I was predicting that Stockfish NNUE SV 1705 would break record at this time control (2 min + 0.5 sec), because this net scales with time much better than other SV networks but the performance of Stockfish NNUE SV 1705 was above my guess.

[mehmet123]

SV 20200824-1705 vs. SV 20200812-2257 (Default Net)

Program Elo + - Games Score Av.Op. Draws

1 Stockfish NNUE SV 1705 : 2401 10 9 660 50.2 % 2399 87.1 %
2 Stockfish NNUE SV 2257 : 2399 9 10 660 49.8 % 2401 87.1 %

Individual statistics:

1 Stockfish NNUE SV 1705 : 2401 660 (+ 44,=575,- 41), 50.2 %

Stockfish NNUE SV 2257 : 660 (+ 44,=575,- 41), 50.2 %

2 Stockfish NNUE SV 2257 : 2399 660 (+ 41,=575,- 44), 49.8 %

Stockfish NNUE SV 1705 : 660 (+ 41,=575,- 44), 49.8 %


Game Conditions: Cutechess Gui, 1 Core ( i7 9750h), 2 min + 0.5 sec TC, Balsa 5 Move Opening Book, 256 Mb Hash, Ponder Off
Compilation: Stockfish NNUE 240820 x64 BMI2 mingw10.2 (ChessMan compile)
http://www.mediafire.com/file/rzxk9m5wj ... 0.pgn/file

According to one to one match Stockfish NNUE SV 1705 is a very little strong (+2 elo) according to Stockfish NNUE SV 2257 (playing with default net) at 2 min + 0.5 sec time control but the result is in the error bars.
Stockfish NNUE SV 1705 seems to have a more aggressive playing style when looking at its score against Stockfish Dev (+128 elo) at 2 min + 0.5 sec time control
SV 1705 is the first net trained from scratch by Sergio Vieri so this also makes me hope for new nets to come.

[mehmet123]

Cfish vs Stockfish (Fischer Random Match)

Program Elo + - Games Score Av.Op. Draws
1 Cfish 250820 x64 bmi2 : 2410 16 16 500 52.8 % 2390 72.4 %
2 Stockfish 260820 x64 bmi2 : 2390 16 16 500 47.2 % 2410 72.4 %

Individual statistics:

1 Cfish 250820 x64 bmi2 : 2410 500 (+ 83,=362,- 55), 52.8 %

Stockfish 260820 x64 bmi2 : 500 (+ 83,=362,- 55), 52.8 %

2 Stockfish 260820 x64 bmi2 : 2390 500 (+ 55,=362,- 83), 47.2 %

Cfish 250820 x64 bmi2 : 500 (+ 55,=362,- 83), 47.2 %


Game Conditions: Cutechess Gui, 1 Core (i7 9750h), 30 sec + 0.5 sec TC, Chess960 Book 3moves, 128 Mb Hash, Ponder Off
Compilation1: Cfish 250820 EXT x64 BMI2 ((ChessMan (Member of The Outskirt Chess Forum) compile )) // Default Net
Compilation2: Stockfish 260820 x64 bmi2 (Official compile)// Default Net
http://www.mediafire.com/file/tzg07ewaw ... 2.pgn/file

Great result. Cfish manages to beat Stockfish Dev without difficulty (+20 elo ) at Fischer Random Match.

[mehmet123]

SV 20200824-1705 vs. SV 20200812-2257 (Default Net)
(Fischer Random Chess Match)

Program Elo + - Games Score Av.Op. Draws

1 Stockfish NNUE SV 1705 : 2402 12 12 1000 50.5 % 2398 66.9 %
2 Stockfish NNUE SV 2257 : 2398 12 12 1000 49.5 % 2402 66.9 %


Individual statistics:

1 Stockfish NNUE SV 1705 : 2402 1000 (+171,=669,-160), 50.5 %

Stockfish NNUE SV 2257 : 1000 (+171,=669,-160), 50.5 %

2 Stockfish NNUE SV 2257 : 2398 1000 (+160,=669,-171), 49.5 %

Stockfish NNUE SV 1705 : 1000 (+160,=669,-171), 49.5 %


Game Conditions: Cutechess Gui, 1 Core (i7 9750h), 30 sec + 0.5 sec TC, Chess960 Book 3moves, 128 Mb Hash, Ponder Off
Compilation1: Stockfish NNUE 240820 x64 BMI2 mingw10.2 (ChessMan compile)
http://www.mediafire.com/file/yv58m7nsb ... 3.pgn/file

According to this test Stockfish NNUE SV 1705 is a little stronger (+4 elo) than Stockfish NNUE SV 2257 (playing with default net) at Fischer Random Chess match but the result is in the error bars.

[mehmet123]

Cfish vs Stockfish Match:

Program Elo + - Games Score Av.Op. Draws

1 Cfish 290820 x64 bmi2 : 2404 6 6 2000 51.1 % 2396 86.6 %
2 Stockfish 290820 x64 bmi2 : 2396 6 6 2000 48.9 % 2404 86.6 %

Individual statistics:

1 Cfish 290820 x64 bmi2 : 2404 2000 (+156,=1732,-112), 51.1 %

Stockfish 290820 x64 bmi2 : 2000 (+156,=1732,-112), 51.1 %

2 Stockfish 290820 x64 bmi2 : 2396 2000 (+112,=1732,-156), 48.9 %

Cfish 290820 x64 bmi2 : 2000 (+112,=1732,-156), 48.9 %


Game Conditions: Cutechess Gui, 1 Core (i7 9750h), 30 sec + 0.5 sec TC, Balsa 5 Move Opening Book, 128 Mb Hash, Ponder Off
Compilation1: Cfish 290820 x64 BMI2 popcnt mingw 10.2 ((ChessMan (Member of The Outskirt Chess Forum) compile )) // Default Net
Compilation2: Stockfish 290820 x64 bmi2 (Official compile)// Default Net
http://www.mediafire.com/file/j1kj15o68 ... 0.pgn/file

Cfish manages to beat the Latest Stockfish Dev as in all other tests.

[mehmet123]

SV 20200903-1739 vs. SV 20200812-2257 (Default Net)

Program Elo + - Games Score Av.Op. Draws

1 Stockfish 12 x64 bmi2 (1739) : 2404 8 8 1200 51.1 % 2396 84.9 %
2 Stockfish 12 x64 bmi2 (2257) : 2396 8 8 1200 48.9 % 2404 84.9 %

Individual statistics:

1 Stockfish 12 x64 bmi2 (1739): 2404 1200 (+104,=1019,- 77), 51.1 %

Stockfish 12 x64 bmi2 (2257) : 1200 (+104,=1019,- 77), 51.1 %

2 Stockfish 12 x64 bmi2 (2257): 2396 1200 (+ 77,=1019,-104), 48.9 %

Stockfish 12 x64 bmi2 (1739) : 1200 (+ 77,=1019,-104), 48.9 %


Game Conditions: Cutechess Gui, 1 Core (i7 9750h), 30 sec + 0.5 sec TC, Balsa 5 Move Opening Book, 128 Mb Hash, Ponder Off
Compilation:Official Version
http://www.mediafire.com/file/w3r0g1y29 ... 7.pgn/file

The performance of latest SV Net is very good at this time control (30 sec + 0.5 sec).
This is the best result (+8 elo) against Stockfish (Default Net) in my tests.
The result of SV 1705 vs. SV 2257 (Default Net) was +2 elo at 1 min 0.5 time control.

[perejaslav]

Hi, Mehmet!

Can you upload all your games from this thread in one pgn file or one zip?

[mehmet123]

Hi, Mehmet!

Can you upload all your games from this thread in one pgn file or one zip?

Anyone else can do that. I don't think anyone will need me on this.
But there is some problem to make a rating list with all the played games. Making a rating list from all the games played can lead to erroneous conclusions because of some scaling problems of SV Nets.
I have making different time control tests. Min:1 core, 30 sec + 0.5 sec , Max:6 core, 2 min + 0.5 sec.
The performance of SV nets at 30 sec + 0.5 sec tc are more better than at 1 min + 0.5 tc.
The performance of SV nets at 1 min + 0.5 sec tc are more better than at 2 min + 0.5 tc.