Sergio Vieri second net is out

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Zenmastur
Posts: 898
Joined: Sat May 31, 2014 6:28 am

Re: Sergio Vieri second net is out

Post by Zenmastur » Sat Aug 01, 2020 8:39 am

Laskos wrote:
Sat Aug 01, 2020 8:28 am
It seems today's net 1209 is the best with 7 +/- 6 Elo points in 8000 games over the past best net.

Code: Select all

Score of Sergio_1209 vs Sergio_29.07_0109: 1180 - 1073 - 1747  [0.513] 4000
...      Sergio_NNUE playing White: 900 - 266 - 834  [0.658] 2000
...      Sergio_NNUE playing Black: 280 - 807 - 913  [0.368] 2000
...      White vs Black: 1707 - 546 - 1747  [0.645] 4000
Elo difference: 9.3 +/- 8.1, LOS: 98.8 %, DrawRatio: 43.7 %
Finished match

Score of Sergio_1209 vs Sergio_29.07_0109: 1195 - 1145 - 1660  [0.506] 4000
...      Sergio_NNUE playing White: 908 - 307 - 785  [0.650] 2000
...      Sergio_NNUE playing Black: 287 - 838 - 875  [0.362] 2000
...      White vs Black: 1746 - 594 - 1660  [0.644] 4000
Elo difference: 4.3 +/- 8.2, LOS: 84.9 %, DrawRatio: 41.5 %
Finished match
I'm about 3,000 games into a 4,000 game match of 0252 vs 1209 and the results seem to echo your results very nicely.

I've noticed there hasn't been much progress since 2138 until the last two nets. Has he changed his search depth?

Regards,

Zenmastur
Only 2 defining forces have ever offered to die for you.....Jesus Christ and the American Soldier. One died for your soul, the other for your freedom.

JohnS
Posts: 172
Joined: Sun Feb 24, 2008 1:08 am

Re: Sergio Vieri second net is out

Post by JohnS » Sat Aug 01, 2020 9:18 am

Laskos wrote:
Sat Aug 01, 2020 8:28 am
It seems today's net 1209 is the best with 7 +/- 6 Elo points in 8000 games over the past best net.

Code: Select all

Score of Sergio_1209 vs Sergio_29.07_0109: 1180 - 1073 - 1747  [0.513] 4000
...      Sergio_NNUE playing White: 900 - 266 - 834  [0.658] 2000
...      Sergio_NNUE playing Black: 280 - 807 - 913  [0.368] 2000
...      White vs Black: 1707 - 546 - 1747  [0.645] 4000
Elo difference: 9.3 +/- 8.1, LOS: 98.8 %, DrawRatio: 43.7 %
Finished match

Score of Sergio_1209 vs Sergio_29.07_0109: 1195 - 1145 - 1660  [0.506] 4000
...      Sergio_NNUE playing White: 908 - 307 - 785  [0.650] 2000
...      Sergio_NNUE playing Black: 287 - 838 - 875  [0.362] 2000
...      White vs Black: 1746 - 594 - 1660  [0.644] 4000
Elo difference: 4.3 +/- 8.2, LOS: 84.9 %, DrawRatio: 41.5 %
Finished match
Looks like we have a new champ. Sergio's own results don't suggest an improvement over the last few days. Maybe his own tests are not as reliable as those here.

User avatar
Leto
Posts: 2040
Joined: Thu May 04, 2006 1:40 am
Location: Dune

Re: Sergio Vieri second net is out

Post by Leto » Sun Aug 02, 2020 1:31 pm

JohnS wrote:
Sat Aug 01, 2020 9:18 am
Laskos wrote:
Sat Aug 01, 2020 8:28 am
It seems today's net 1209 is the best with 7 +/- 6 Elo points in 8000 games over the past best net.

Code: Select all

Score of Sergio_1209 vs Sergio_29.07_0109: 1180 - 1073 - 1747  [0.513] 4000
...      Sergio_NNUE playing White: 900 - 266 - 834  [0.658] 2000
...      Sergio_NNUE playing Black: 280 - 807 - 913  [0.368] 2000
...      White vs Black: 1707 - 546 - 1747  [0.645] 4000
Elo difference: 9.3 +/- 8.1, LOS: 98.8 %, DrawRatio: 43.7 %
Finished match

Score of Sergio_1209 vs Sergio_29.07_0109: 1195 - 1145 - 1660  [0.506] 4000
...      Sergio_NNUE playing White: 908 - 307 - 785  [0.650] 2000
...      Sergio_NNUE playing Black: 287 - 838 - 875  [0.362] 2000
...      White vs Black: 1746 - 594 - 1660  [0.644] 4000
Elo difference: 4.3 +/- 8.2, LOS: 84.9 %, DrawRatio: 41.5 %
Finished match
Looks like we have a new champ. Sergio's own results don't suggest an improvement over the last few days. Maybe his own tests are not as reliable as those here.
Do you mean the rating graph on Sergio's net download page? I think that rating is done with games played at depth 16 per move, if so then yes it wouldn't be very reliable, probably more of a quick check to see if anything gets broken.

User avatar
Sylwy
Posts: 3370
Joined: Fri Apr 21, 2006 2:19 pm
Location: IASI (Romania) - the historical capital of MOLDOVA

Re: Sergio Vieri second net is out

Post by Sylwy » Sun Aug 02, 2020 4:38 pm

A comparative test: SV 2141 vs. SV 2138.

Both tests in the same conditions:

-TC=4'+2"
-Hash=256 MB
-GUI: Arena 3.5.1
-Book: IM_4mvs.abk (8 plies book) for both engines
-default settings for both engines
-1 thread-CPU=Intel i5-7400-3GHz (Kaby Lake)
-6-men Syzygy bases for both engines.

The binaries (Mr. Hisayori Noda) were 2020-07-19 BMI2 version.

Image

Image

Image

Image

I'm sure that SF-NNUE - together with SV 2138 net - has over 3.650 Elo (CCRL-Blitz-1 CPU conditions).
SilvianR :wink:

JohnS
Posts: 172
Joined: Sun Feb 24, 2008 1:08 am

Re: Sergio Vieri second net is out

Post by JohnS » Mon Aug 03, 2020 1:31 am

Leto wrote:
Sun Aug 02, 2020 1:31 pm

Do you mean the rating graph on Sergio's net download page? I think that rating is done with games played at depth 16 per move, if so then yes it wouldn't be very reliable, probably more of a quick check to see if anything gets broken.
Yes the graph on his website was what I was thinking of.

User avatar
Rebel
Posts: 5349
Joined: Thu Aug 18, 2011 10:04 am

Re: Sergio Vieri second net is out

Post by Rebel » Mon Aug 03, 2020 3:58 am

Got a new champion, Sergio-2257, 64.1%, ~99 elo

Code: Select all

sergio-0109   63.6%   2020-07-29 01:09 | 5000 games
sergio-2214   62.9%   2020-07-29 22:15 | 5000 games
sergio-2138   63.4%   2020-07-28 23:44 | 5000 games
sergio-0111   62.4%   2020-07-31 01:11 | 5000 games
sergio-1209   63.4%   2020-08-01 12:21 | 5000 games
sergio-1515   62.3%   2020-08-01 19:35 | 5000 games
sergio-2257   64.1%   2020-08-02 23:13 | 5000 games
For a long time it had a score of 65% but in the last 1000 games it settled for 64.1%

Edit - will put this version (ready to use) on my website installed in Arena.
90% of coding is debugging, the other 10% is writing bugs.

JohnS
Posts: 172
Joined: Sun Feb 24, 2008 1:08 am

Re: Sergio Vieri second net is out

Post by JohnS » Mon Aug 03, 2020 4:13 am

Rebel wrote:
Mon Aug 03, 2020 3:58 am
Got a new champion, Sergio-2257, 64.1%, ~99 elo

Code: Select all

sergio-0109   63.6%   2020-07-29 01:09 | 5000 games
sergio-2214   62.9%   2020-07-29 22:15 | 5000 games
sergio-2138   63.4%   2020-07-28 23:44 | 5000 games
sergio-0111   62.4%   2020-07-31 01:11 | 5000 games
sergio-1209   63.4%   2020-08-01 12:21 | 5000 games
sergio-1515   62.3%   2020-08-01 19:35 | 5000 games
sergio-2257   64.1%   2020-08-02 23:13 | 5000 games
For a long time it had a score of 65% but in the last 1000 games it settled for 64.1%

Edit - will put this version (ready to use) on my website installed in Arena.
Wow that was quick :) The champs have a very short reign and who can keep up all the new challengers.

User avatar
MikeB
Posts: 4181
Joined: Thu Mar 09, 2006 5:34 am
Location: Pen Argyl, Pennsylvania

Re: Sergio Vieri second net is out

Post by MikeB » Mon Aug 03, 2020 6:05 am

Rebel wrote:
Mon Aug 03, 2020 3:58 am
Got a new champion, Sergio-2257, 64.1%, ~99 elo

Code: Select all

sergio-0109   63.6%   2020-07-29 01:09 | 5000 games
sergio-2214   62.9%   2020-07-29 22:15 | 5000 games
sergio-2138   63.4%   2020-07-28 23:44 | 5000 games
sergio-0111   62.4%   2020-07-31 01:11 | 5000 games
sergio-1209   63.4%   2020-08-01 12:21 | 5000 games
sergio-1515   62.3%   2020-08-01 19:35 | 5000 games
sergio-2257   64.1%   2020-08-02 23:13 | 5000 games
For a long time it had a score of 65% but in the last 1000 games it settled for 64.1%

Edit - will put this version (ready to use) on my website installed in Arena.
I also saw a nice gain with 2257:

Code: Select all

$ run
Settings ...

Current date : time (EDST)
Date: 08/02/20 : 03:09:48

CMD1=Stockfish-XI-NN.exe ENG1=St-0252 EVFILE1=./eval/20200731-0252.bin
CMD2=Stockfish-XI-NN.exe ENG2=St-1515 EVFILE2=./eval/20200801-1515.bin
CMD4=Black-Diamond-XI-NN.exe ENG4=BD-0252 EVFILE4=./eval/20200731-0252.bin

PGN File: c:/cluster.mfb/pgn/08020309.pgn
Games: 4000
Projected-> Time: 4h:22m:27s
50+1.00

Time Control-> base+inc: 50+1.00
Start running the chess match ...
/c/Users/MichaelB7/home/Github/cutechess/projects/cli/run: line 208: 2nd: command not found

#########################################################################################################
###                                              Summary                                              ###
#########################################################################################################

PGN File: c:/cluster.mfb/pgn/08020309.pgn
Time Control: Time Control-> base+inc: 50+1.00
Games: 4000
Threads: 1
Hash: 256

Current date : time (EDST)
Date: 08/02/20 : 09:25:28

Projected-> Time: 4h:22m:27s
     Run -> Time: 6h:15m:40s

6000 game(s) loaded
Rank Name     Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR
---------------------------------------------------------------------------------------------------------

   1 St-1515   3504   0.0    4    4  6000 3061.0  51.0  681  559 4760  11.3  79.3  3496
   2 St-0252   3496   7.0    4    4  6000 2939.0  49.0  559  681 4760   9.3  79.3  3504
---------------------------------------------------------------------------------------------------------

  Δ = delta from the next higher rated opponent
  # = number of games played
  Σ = total score, 1 point for win, 1/2 point for draw

LOS:
         St St
St-1515     99
St-0252   0

#########################################################################################################
###                                                End                                                ###
#########################################################################################################


6000 game(s) loaded
Speaking of 2257 < I have modified Honey to so you can run a bench with any net that you desired - as you may or may not know, every bin has a different bench signature
format is the normal bench with two additional parameters - one tha NN is true the name and location of file
Ho*r4*.exe b 16 1 13 true C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin >/dev/null

Code: Select all

Ho*r4*.exe  b 16 1 13 true C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin >/dev/null

Position: 1/45
Nodes/Second: 1182486
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 2/45
Nodes/Second: 1266516
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 3/45
Nodes/Second: 1813785
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 4/45
Nodes/Second: 1312303
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 5/45
Nodes/Second: 1170134
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 6/45
Nodes/Second: 1191088
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 7/45
Nodes/Second: 1220884
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 8/45
Nodes/Second: 1303099
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 9/45
Nodes/Second: 1215618
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 10/45
Nodes/Second: 1314588
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 11/45
Nodes/Second: 1231566
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 12/45
Nodes/Second: 1260754
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 13/45
Nodes/Second: 1325046
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 14/45
Nodes/Second: 1269061
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 15/45
Nodes/Second: 1414901
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 16/45
Nodes/Second: 1309956
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 17/45
Nodes/Second: 1429676
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 18/45
Nodes/Second: 1418933
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 19/45
Nodes/Second: 1848273
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 20/45
Nodes/Second: 1689525
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 21/45
Nodes/Second: 1968583
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 22/45
Nodes/Second: 2136333
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 23/45
Nodes/Second: 2351380
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 24/45
Nodes/Second: 1506656
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 25/45
Nodes/Second: 1413363
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 26/45
Nodes/Second: 1574833
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 27/45
Nodes/Second: 1466844
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 28/45
Nodes/Second: 1347375
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 29/45
Nodes/Second: 1395311
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 30/45
Nodes/Second: 1713045
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 31/45
Nodes/Second: 1425828
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 32/45
Nodes/Second: 1325442
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 33/45
Nodes/Second: 1246672
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 34/45
Nodes/Second: 1219804
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 35/45
Nodes/Second: 1581569
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 36/45
Nodes/Second: 1970937
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 37/45
Nodes/Second: 2260260
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 38/45
Nodes/Second: 2186288
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 39/45
Nodes/Second: 1901694
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 40/45
Nodes/Second: 1954192
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 41/45
Nodes/Second: 2242615
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 42/45
Nodes/Second: 1810852
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 43/45
Nodes/Second: 1121000
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 44/45
Nodes/Second: 1181090
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 45/45
Nodes/Second: 1147764
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

=================================
Total time (ms) : 3285
Nodes searched  : 4732771

Nodes/second    : 1440721
and lastly , you will be able to switch back and forth easily from the command line
Image
right click and select open in new window to a more readable version

hit the return key to get current status"
"s nn true" to enable NN evaluation
run your search
"s nn false" enable classical evaluation
run your search
literally two engines at your disposal while running one exe
Image

peter
Posts: 2004
Joined: Sat Feb 16, 2008 6:38 am
Full name: Peter Martan

Re: Sergio Vieri second net is out

Post by peter » Mon Aug 03, 2020 6:44 am

MikeB wrote:
Mon Aug 03, 2020 6:05 am

and lastly , you will be able to switch back and forth easily from the command line
Image
right click and select open in new window to a more readable version

hit the return key to get current status"
"s nn true" to enable NN evaluation
run your search
"s nn false" enable classical evaluation
run your search
literally two engines at your disposal while running one exe
Sounds great, Mike, will there be other compiles except the one you already made available?
I know, classical popcnt isn't really good for NNUE, but there are some kinds of "extra- popcnt" around. "Legacy", "ssse3-popcnt", "AIIAVXpopcnt" are some of the names, that did work with my old Xeon X5670.
Peter.

Milos
Posts: 3762
Joined: Wed Nov 25, 2009 12:47 am

Re: Sergio Vieri second net is out

Post by Milos » Mon Aug 03, 2020 7:11 am

peter wrote:
Mon Aug 03, 2020 6:44 am
MikeB wrote:
Mon Aug 03, 2020 6:05 am

and lastly , you will be able to switch back and forth easily from the command line
Image
right click and select open in new window to a more readable version

hit the return key to get current status"
"s nn true" to enable NN evaluation
run your search
"s nn false" enable classical evaluation
run your search
literally two engines at your disposal while running one exe
Sounds great, Mike, will there be other compiles except the one you already made available?
I know, classical popcnt isn't really good for NNUE, but there are some kinds of "extra- popcnt" around. "Legacy", "ssse3-popcnt", "AIIAVXpopcnt" are some of the names, that did work with my old Xeon X5670.
The actual NNUE implementation is handcrafted and only has AVX2, SSE2 and SSE4.1 implementation. So no additional compiler switches would make any difference at all.

Post Reply