Stockfish NNUE SV Tests

Discussion of computer chess matches and engine tournaments.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
carldaman
Posts: 2075
Joined: Sat Jun 02, 2012 12:13 am

Re: Stockfish NNUE SV Tests

Post by carldaman » Sat Dec 19, 2020 2:36 am

mehmet123 wrote:
Fri Dec 18, 2020 4:43 pm

Game Conditions: Cutechess Gui, 1 Core (Core-i7 9750h), 2 min + 0.5 sec TC, Balsa 5 Moves Opening Book, 128 Mb Hash, Ponder Off//Default Net
Cfish version:Cfish 131220 x64 E BMI2 mingw10 (ChessMan version)
https://www.mediafire.com/file/phz4bvdd ... 5.pgn/file

This is a different experience file than Exp3 file. They are basically close files but there are some differences in game database and settings. Minimum experience depth is 16 for this file. For Exp3 file the game database is smaller than V16 file and minimum experience depth is 18.
Too many games in experience file does not mean the file will be stronger. Sometimes ineffective games weaken the experience file.
Interesting observations, but what do you mean by ''ineffective games" ?

Shouldn't the learning process weed out the bad moves or lines, anyway?
If such games weaken the exp file, then it would be a sign of ineffective learning, would it not?

But don't get me wrong, I very much like and support all these engines and authors who have implemented learning.
I hope they succeed in their mission. :)

mehmet123
Posts: 292
Joined: Sun Jan 26, 2020 9:38 pm
Location: Turkey
Full name: Mehmet Karaman

Re: Stockfish NNUE SV Tests

Post by mehmet123 » Sat Dec 19, 2020 1:36 pm

carldaman wrote:
Sat Dec 19, 2020 2:36 am

Interesting observations, but what do you mean by ''ineffective games" ?

Shouldn't the learning process weed out the bad moves or lines, anyway?
If such games weaken the exp file, then it would be a sign of ineffective learning, would it not?

But don't get me wrong, I very much like and support all these engines and authors who have implemented learning.
I hope they succeed in their mission. :)
This isn't about same experience file. I had prepared an experience file and I have testing this file by merging some other experience files. Some of the experience files don't contribute to a good performance after this merge for my test conditions. Of course, each experience file with contains sufficient number of games will contribute more or less to the chess engine.

carldaman
Posts: 2075
Joined: Sat Jun 02, 2012 12:13 am

Re: Stockfish NNUE SV Tests

Post by carldaman » Sun Dec 20, 2020 2:35 am

OK, thanks for the clarification. :)

mehmet123
Posts: 292
Joined: Sun Jan 26, 2020 9:38 pm
Location: Turkey
Full name: Mehmet Karaman

Re: Stockfish NNUE SV Tests

Post by mehmet123 » Tue Dec 22, 2020 10:03 pm

Sugar AI vs Cfish:

Program Elo + - Games Score Av.Op. Draws

1 Cfish 171220 x64 bmi2 : 2404 11 10 300 51.2 % 2396 92.3 %
2 SugaR AI 1.20 bmi2 : 2396 10 11 300 48.8 % 2404 92.3 %


Individual statistics:

1 Cfish 171220 x64 bmi2 : 2404 300 (+ 15,=277,- 8), 51.2 %

SugaR AI 1.20 bmi2 : 300 (+ 15,=277,- 8), 51.2 %

2 SugaR AI 1.20 bmi2 : 2396 300 (+ 8,=277,- 15), 48.8 %

Cfish 171220 x64 bmi2 : 300 (+ 8,=277,- 15), 48.8 %


Game Conditions: Cutechess Gui, 1 Core (Core-i7 9750h), 1 min + 0.5 sec TC, Balsa 5 Moves Opening Book, 128 Mb Hash, Ponder Off
Cfish compile:Cfish 171220 x64 E BMI2 mingw10 (Chessman compile) // Sugar AI wasn't use experince file//Default Nets
https://www.mediafire.com/file/it8qr4uy ... 2.pgn/file

mehmet123
Posts: 292
Joined: Sun Jan 26, 2020 9:38 pm
Location: Turkey
Full name: Mehmet Karaman

Re: Stockfish NNUE SV Tests

Post by mehmet123 » Tue Dec 22, 2020 10:03 pm

SugaR AI v Cfish:

Program Elo + - Games Score Av.Op. Draws

1 SugaR AI 1.00 x64 bmi2 TX1 : 2401 7 7 420 50.2 % 2399 95.2 %
2 Cfish 131220 x64 bmi2 : 2399 7 7 420 49.8 % 2401 95.2 %


Individual statistics:

1 SugaR AI 1.00 x64 bmi2 TX1: 2401 420 (+ 11,=400,- 9), 50.2 %

Cfish 131220 x64 bmi2 : 420 (+ 11,=400,- 9), 50.2 %

2 Cfish 131220 x64 bmi2 : 2399 420 (+ 9,=400,- 11), 49.8 %

SugaR AI 1.00 x64 bmi2 TX1 : 420 (+ 9,=400,- 11), 49.8 %


Game Conditions: Cutechess Gui, 1 Core (Core-i7 9750h), 5 min TC, Balsa 5 Moves Opening Book, 128 Mb Hash, Ponder Off
Cfish compile:Cfish 131220 x64 E BMI2 mingw10 (Chessman compile) // Sugar AI played with TX1 exp file//Default Nets
https://www.mediafire.com/file/b53zrwc6 ... 1.pgn/file

The experience files I used in my previous tests were very large compared to these files. They have a value between 100 MB and 1 GB but this experience file (TX1) is very small compared to other files, although it is only 8.5 MB. Sugar AI with this exp file managed to perform well in the match under these conditions( 5 minutes/game)

mehmet123
Posts: 292
Joined: Sun Jan 26, 2020 9:38 pm
Location: Turkey
Full name: Mehmet Karaman

Re: Stockfish NNUE SV Tests

Post by mehmet123 » Wed Dec 30, 2020 7:55 pm

SugaR AI vs Stockfish (Fischer Random Chess Match)

Program Elo + - Games Score Av.Op. Draws

1 SugaR AI 1.20 x64 bmi2 : 2403 12 12 1000 51.0 % 2397 69.0 %
2 Stockfish 251220 x64 bmi2 : 2397 12 12 1000 49.0 % 2403 69.0 %

Individual statistics:

1 SugaR AI 1.20 x64 bmi2 : 2403 1000 (+165,=690,-145), 51.0 %

Stockfish 251220 x64 bmi2 : 1000 (+165,=690,-145), 51.0 %

2 Stockfish 251220 x64 bmi2 : 2397 1000 (+145,=690,-165), 49.0 %

SugaR AI 1.20 x64 bmi2 : 1000 (+145,=690,-165), 49.0 %


Game Conditions: Cutechess Gui, 1 Core ( i7 9750h), 10 sec + 0.2 sec TC, Chess960 Book 3 Moves, 64 Mb Hash, Ponder Off
Default Nets// SugaR AI played without experience file
https://www.mediafire.com/file/40wvzfv8 ... 8.pgn/file

mehmet123
Posts: 292
Joined: Sun Jan 26, 2020 9:38 pm
Location: Turkey
Full name: Mehmet Karaman

Re: Stockfish NNUE SV Tests

Post by mehmet123 » Wed Jan 06, 2021 7:20 pm

SugaR AI vs Stockfish:

Program Elo + - Games Score Av.Op. Draws

1 SugaR AI 1.30 x64 bmi2 : 2401 7 7 1000 50.1 % 2399 88.7 %
2 Stockfish 311220 x64 bmi2 : 2399 7 7 1000 49.9 % 2401 88.7 %

Individual statistics:

1 SugaR AI 1.30 x64 bmi2 : 2401 1000 (+ 58,=887,- 55), 50.1 %

Stockfish 311220 x64 bmi2 : 1000 (+ 58,=887,- 55), 50.1 %

2 Stockfish 311220 x64 bmi2 : 2399 1000 (+ 55,=887,- 58), 49.9 %

SugaR AI 1.30 x64 bmi2 : 1000 (+ 55,=887,- 58), 49.9 %


Game Conditions: Cutechess Gui, 1 Core (Core-i7 9750h), 30 sec + 0.5 sec TC, Balsa 5 Moves Opening Book, 64 Mb Hash, Ponder Off
Sugar AI didn' t use experince file//Default Nets
https://www.mediafire.com/file/zmdz0u6t ... 1.pgn/file

mehmet123
Posts: 292
Joined: Sun Jan 26, 2020 9:38 pm
Location: Turkey
Full name: Mehmet Karaman

Re: Stockfish NNUE SV Tests

Post by mehmet123 » Tue Jan 19, 2021 7:59 pm

Cfish vs Sugar AI:

Program Elo + - Games Score Av.Op. Draws

1 Cfish 040121 x64 bmi2 : 2401 4 4 950 50.2 % 2399 96.8 %
2 SugaR AI 1.40 bmi2 : 2399 4 4 950 49.8 % 2401 96.8 %

Individual statistics:

1 Cfish 040121 x64 bmi2 : 2401 950 (+ 17,=920,- 13), 50.2 %

SugaR AI 1.40 bmi2 : 950 (+ 17,=920,- 13), 50.2 %

2 SugaR AI 1.40 bmi2 : 2399 950 (+ 13,=920,- 17), 49.8 %

Cfish 040121 x64 bmi2 : 950 (+ 13,=920,- 17), 49.8 %


Game Conditions: Cutechess Gui, 1 Core (Core-i7 9750h), 5 min TC, Balsa 5 Moves Opening Book, 512 Mb Hash, Ponder Off
Cfish Version: Cfish Ext 040121 x64 ELTO BMI2 (ChessMan version)
Sugar AI 1.40 didn' t use experince file//Default Nets
https://www.mediafire.com/file/agnwvsqp ... 2.pgn/file

This is the best performance of any chess engine against Cfish in my tests. Sugar AI 1.00 had beaten Cfish 131220 in my previous test, but at that time Sugar AI was using learning file.

mehmet123
Posts: 292
Joined: Sun Jan 26, 2020 9:38 pm
Location: Turkey
Full name: Mehmet Karaman

Re: Stockfish NNUE SV Tests

Post by mehmet123 » Sun Jan 31, 2021 7:09 am

Eman vs Stockfish:

Program Elo + - Games Score Av.Op. Draws

1 Eman 6.90 x64 bmi2 : 2403 7 7 1000 50.7 % 2397 90.5 %
2 Stockfish 300121 x64 bmi2 : 2397 7 7 1000 49.2 % 2403 90.5 %

Individual statistics:

1 Eman 6.90 x64 bmi2 : 2403 1000 (+ 55,=905,- 40), 50.8 %

Stockfish 300121 x64 bmi2 : 1000 (+ 55,=905,- 40), 50.7 %

2 Stockfish 300121 x64 bmi2 : 2397 1000 (+ 40,=905,- 55), 49.2 %

Eman 6.90 x64 bmi2 : 1000 (+ 40,=905,- 55), 49.2 %


Game Conditions: Cutechess Gui, 1 Core (Core-i7 9750h), 30 sec + 0.5 sec TC, Balsa 5 Moves Opening Book, 128 Mb Hash, Ponder Off
Eman 6.90 didn' t use experince file//Default Nets
https://www.mediafire.com/file/dirh5l7h ... 3.pgn/file

mehmet123
Posts: 292
Joined: Sun Jan 26, 2020 9:38 pm
Location: Turkey
Full name: Mehmet Karaman

Re: Stockfish NNUE SV Tests

Post by mehmet123 » Tue Feb 02, 2021 3:28 pm

15 years change of chess engines :

Program Elo + - Games Score Av.Op. Draws

1 Stockfish 310121 x64 bmi2 : 1040 0 0 200 99.8 % 2100 0.5 %
2 Rybka 1.0 Beta x64 : 0 0 0 200 0.2 % 2700 0.5 %

Individual statistics:

1 Stockfish 310121 x64 bmi2 : 1040 200 (+199,= 1,- 0), 99.8 %

Rybka 1.0 Beta x64 : 200 (+199,= 1,- 0), 99.8 %

2 Rybka 1.0 Beta x64 : 0 200 (+ 0,= 1,-199), 0.2 %

Stockfish 310121 x64 bmi2 : 200 (+ 0,= 1,-199), 0.2 %


Game Conditions: Cutechess Gui, 1 Core ( i7 9750h), 1 min + 0.5 sec TC, Balsa 5 Move Opening Book, 128 Mb Hash, Ponder Off
Stockfish 310121 x64 bmi2 played with default nnue and contempt:60 setting
https://www.mediafire.com/file/a5ndxd30 ... 4.pgn/file

Rybka 1.0 x64 was the most powerful chess engine as of February 2006. Rybka 1.0 Beta x64 is +53 elo stronger than Fritz 10 at 1 cpu (Cegt 40/20). Deep Fritz 10 defeated world chess champion Kramnik in 2005 with a score of 4-2.

Post Reply