Page 2 of 27
Re: Stockfish NNUE SV Tests
Posted: Thu Jul 23, 2020 10:11 pm
by mehmet123
The performance of Stockfish NNUE SV Exp 20200722-1130 net against Stockfish 170720 x64 bmi2
Total 1025 games with Balsa 5 move opening book
3 different time controls ( 30 sec +0. 5 sec, 1 min + 0.5 sec, 2 min + 0.5 sec)
Average elo difference is ~58 elo
For now this is a record for my tests.
Re: Stockfish NNUE SV Tests
Posted: Fri Jul 24, 2020 4:54 am
by mehmet123
Test of SV 20200724-0123 Net
Program Elo + - Games Score Av.Op. Draws
1 Stockfish NNUE 190720 bmi2 :
2430 16 16 600 58.6 % 2370 64.8 %
2 Stockfish 17072020 x64 bmi2 :
2370 16 16 600 41.4 % 2430 64.8 %
Individual statistics:
1 Stockfish NNUE 190720 bmi2:
2430 600 (+157,=389,- 54), 58.6 %
Stockfish 17072020 x64 bmi2 : 600 (+157,=389,- 54), 58.6 %
2 Stockfish 17072020 x64 bmi2 :
2370 600 (+ 54,=389,-157), 41.4 %
Stockfish NNUE 190720 bmi2 : 600 (+ 54,=389,-157), 41.4 %
Cutechess Gui, 1 Core (i7 9750h), 30 sec + 0.5 sec TC, Balsa 5 Move Opening Book, 64 Mb Hash, Ponder Off
http://www.mediafire.com/file/dfgw4zw5j ... 1.pgn/file
Re: Stockfish NNUE SV Tests
Posted: Fri Jul 24, 2020 8:51 am
by carldaman
mehmet123 wrote: ↑Thu Jul 23, 2020 10:11 pm
The performance of Stockfish NNUE SV Exp 20200722-1130 net against Stockfish 170720 x64 bmi2
Total 1025 games with Balsa 5 move opening book
3 different time controls ( 30 sec +0. 5 sec, 1 min + 0.5 sec, 2 min + 0.5 sec)
Average elo difference is
~58 elo
For now this is a record for my tests.
So, are you saying this experimental net outperformed the regular SV nnue nets?
Re: Stockfish NNUE SV Tests
Posted: Fri Jul 24, 2020 9:16 am
by mehmet123
Yes, for now it's the strongest Stockfish NNUE net according to my tests.
But evertyhing can change very fast. Because new SV nets are published at regular time intervals.
Re: Stockfish NNUE SV Tests
Posted: Fri Jul 24, 2020 9:45 am
by carldaman
Interesting that the experimental net is the best, at least for now. Thanks for doing this testing - especially with possible regressions lurking, it's hard to know if the latest is also the best.
Re: Stockfish NNUE SV Tests
Posted: Fri Jul 24, 2020 9:28 pm
by mehmet123
Test of SV 20200724-2344 Net
Program Elo + - Games Score Av.Op. Draws
1 Stockfish NNUE 190720 bmi2 :
2442 24 24 360 61.9 % 2358 55.6 %
2 Stockfish 17072020 x64 bmi2 :
2358 24 24 360 38.1 % 2442 55.6 %
Individual statistics:
1 Stockfish NNUE 190720 bmi2:
2442 360 (+123,=200,- 37), 61.9 %
Stockfish 17072020 x64 bmi2 : 360 (+123,=200,- 37), 61.9 %
2 Stockfish 17072020 x64 bmi2 :
2358 360 (+ 37,=200,-123), 38.1 %
Stockfish NNUE 190720 bmi2 : 360 (+ 37,=200,-123), 38.1 %
Cutechess Gui, 1 Core (i7 9750h), 30 sec + 0.5 sec TC, Balsa 5 Move Opening Book, 16 Mb Hash, Ponder Off
http://www.mediafire.com/file/72p94020j ... 0.pgn/file
Very great result of Stockfish NNUE (20200724-2344). 84 elo difference is a new record for my tests.
Re: Stockfish NNUE SV Tests
Posted: Sat Jul 25, 2020 8:41 am
by mehmet123
Don't forget this these tests has been made with 1 (core i7 9750h) with short time controls (30 sec + 0.5 sec , 1 min + 0.5 sec, 2 min + 0.5).
At long time control matches we shouldn't expect such a big difference elo.
In my tests the average elo performance of SF NNUE Gekehenker is +32 elo against Stockfish Dev. I had made some little test with SF NNUE Gekehenker at 10 minutes time control. I found a difference of + 17 elo performance against Stockfish Dev.
SF NNUE is getting stronger, elo difference will gradually increase in long-term matches.
Re: Stockfish NNUE SV Tests
Posted: Sat Jul 25, 2020 4:13 pm
by mehmet123
Test of SV 20200724-2344 Net
Program Elo + - Games Score Av.Op. Draws
1 Stockfish NNUE SV2344 :
2424 28 26 150 57.0 % 2376 75.3 %
2 Stockfish 170720 x64 bmi2 :
2376 26 28 150 43.0 % 2424 75.3 %
Individual statistics:
1 Stockfish NNUE SV2344 :
2424 150 (+ 29,=113,- 8), 57.0 %
Stockfish 170720 x64 bmi2 : 150 (+ 29,=113,- 8), 57.0 %
2 Stockfish 170720 x64 bmi2 :
2376 150 (+ 8,=113,- 29), 43.0 %
Stockfish NNUE SV2344 : 150 (+ 8,=113,- 29), 43.0 %
Arena Gui, 6 Core (i7 9750h), 2 min + 1 sec TC, Balsa 5 Move Opening Book,512 Mb Hash, Ponder Off
http://www.mediafire.com/file/eoh2keixc ... 0.pgn/file
I tested Stockfish NNUE SV2344 with 6 cores (2min + 1 sec). 48 elo difference is very good for these conditions
Re: Stockfish NNUE SV Tests
Posted: Sat Jul 25, 2020 9:27 pm
by mehmet123
Test of SV 20200723-0511 Net
Program Elo + - Games Score Av.Op. Draws
1 Stockfish NNUE SV 0511 :
2439 20 19 440 60.9 % 2361 62.7 %
2 Stockfish 170720 x64 bmi2 :
2361 19 20 440 39.1 % 2439 62.7 %
Individual statistics:
1 Stockfish NNUE SV 0511 :
2439 440 (+130,=276,- 34), 60.9 %
Stockfish 170720 x64 bmi2 : 440 (+130,=276,- 34), 60.9 %
2 Stockfish 170720 x64 bmi2 :
2361 440 (+ 34,=276,-130), 39.1 %
Stockfish NNUE SV 0511 : 440 (+ 34,=276,-130), 39.1 %
Cutechess Gui, 1 Core (i7 9750h), 30 sec + 0.5 sec TC, Balsa 5 Move Opening Book, 64 Mb Hash, Ponder Off
http://www.mediafire.com/file/utu61s9o8 ... 1.pgn/file
T
Re: Stockfish NNUE SV Tests
Posted: Sun Jul 26, 2020 12:17 pm
by mehmet123
Test of SV 20200723-0511 Net
Program Elo + - Games Score Av.Op. Draws
1 Stockfish NNUE SV 0511 :
2426 26 24 150 57.3 % 2374 78.7 %
2 Stockfish 170720 x64 bmi2 :
2374 24 26 150 42.7 % 2426 78.7 %
Individual statistics:
1 Stockfish NNUE SV 0511 :
2426 150 (+ 27,=118,- 5), 57.3 %
Stockfish 170720 x64 bmi2 : 150 (+ 27,=118,- 5), 57.3 %
2 Stockfish 170720 x64 bmi2 :
2374 150 (+ 5,=118,- 27), 42.7 %
Stockfish NNUE SV 0511 : 150 (+ 5,=118,- 27), 42.7 %
Arena Gui, 6 Core (i7 9750h), 2 min + 1 sec TC, Balsa 5 Move Opening Book,512 Mb Hash, Ponder Off
http://www.mediafire.com/file/e07i9880k ... 1.pgn/file
52 elo difference is a new record at these conditions (6 core , 2 min + 1 sec).