Sergio Vieri second net is out

[Zenmastur]

It seems today's net 1209 is the best with 7 +/- 6 Elo points in 8000 games over the past best net.

Code: Select all

Score of Sergio_1209 vs Sergio_29.07_0109: 1180 - 1073 - 1747  [0.513] 4000
...      Sergio_NNUE playing White: 900 - 266 - 834  [0.658] 2000
...      Sergio_NNUE playing Black: 280 - 807 - 913  [0.368] 2000
...      White vs Black: 1707 - 546 - 1747  [0.645] 4000
Elo difference: 9.3 +/- 8.1, LOS: 98.8 %, DrawRatio: 43.7 %
Finished match

Score of Sergio_1209 vs Sergio_29.07_0109: 1195 - 1145 - 1660  [0.506] 4000
...      Sergio_NNUE playing White: 908 - 307 - 785  [0.650] 2000
...      Sergio_NNUE playing Black: 287 - 838 - 875  [0.362] 2000
...      White vs Black: 1746 - 594 - 1660  [0.644] 4000
Elo difference: 4.3 +/- 8.2, LOS: 84.9 %, DrawRatio: 41.5 %
Finished match

I'm about 3,000 games into a 4,000 game match of 0252 vs 1209 and the results seem to echo your results very nicely.

I've noticed there hasn't been much progress since 2138 until the last two nets. Has he changed his search depth?

Regards,

Zenmastur

[JohnS]

It seems today's net 1209 is the best with 7 +/- 6 Elo points in 8000 games over the past best net.

Code: Select all

Score of Sergio_1209 vs Sergio_29.07_0109: 1180 - 1073 - 1747  [0.513] 4000
...      Sergio_NNUE playing White: 900 - 266 - 834  [0.658] 2000
...      Sergio_NNUE playing Black: 280 - 807 - 913  [0.368] 2000
...      White vs Black: 1707 - 546 - 1747  [0.645] 4000
Elo difference: 9.3 +/- 8.1, LOS: 98.8 %, DrawRatio: 43.7 %
Finished match

Score of Sergio_1209 vs Sergio_29.07_0109: 1195 - 1145 - 1660  [0.506] 4000
...      Sergio_NNUE playing White: 908 - 307 - 785  [0.650] 2000
...      Sergio_NNUE playing Black: 287 - 838 - 875  [0.362] 2000
...      White vs Black: 1746 - 594 - 1660  [0.644] 4000
Elo difference: 4.3 +/- 8.2, LOS: 84.9 %, DrawRatio: 41.5 %
Finished match

Looks like we have a new champ. Sergio's own results don't suggest an improvement over the last few days. Maybe his own tests are not as reliable as those here.

[Leto]

It seems today's net 1209 is the best with 7 +/- 6 Elo points in 8000 games over the past best net.

Code: Select all

Score of Sergio_1209 vs Sergio_29.07_0109: 1180 - 1073 - 1747  [0.513] 4000
...      Sergio_NNUE playing White: 900 - 266 - 834  [0.658] 2000
...      Sergio_NNUE playing Black: 280 - 807 - 913  [0.368] 2000
...      White vs Black: 1707 - 546 - 1747  [0.645] 4000
Elo difference: 9.3 +/- 8.1, LOS: 98.8 %, DrawRatio: 43.7 %
Finished match

Score of Sergio_1209 vs Sergio_29.07_0109: 1195 - 1145 - 1660  [0.506] 4000
...      Sergio_NNUE playing White: 908 - 307 - 785  [0.650] 2000
...      Sergio_NNUE playing Black: 287 - 838 - 875  [0.362] 2000
...      White vs Black: 1746 - 594 - 1660  [0.644] 4000
Elo difference: 4.3 +/- 8.2, LOS: 84.9 %, DrawRatio: 41.5 %
Finished match

Looks like we have a new champ. Sergio's own results don't suggest an improvement over the last few days. Maybe his own tests are not as reliable as those here.

Do you mean the rating graph on Sergio's net download page? I think that rating is done with games played at depth 16 per move, if so then yes it wouldn't be very reliable, probably more of a quick check to see if anything gets broken.

[Sylwy]

A comparative test: SV 2141 vs. SV 2138.

Both tests in the same conditions:

-TC=4'+2"
-Hash=256 MB
-GUI: Arena 3.5.1
-Book: IM_4mvs.abk (8 plies book) for both engines
-default settings for both engines
-1 thread-CPU=Intel i5-7400-3GHz (Kaby Lake)
-6-men Syzygy bases for both engines.

The binaries (Mr. Hisayori Noda) were 2020-07-19 BMI2 version.









I'm sure that SF-NNUE - together with SV 2138 net - has over 3.650 Elo (CCRL-Blitz-1 CPU conditions).

[JohnS]

Do you mean the rating graph on Sergio's net download page? I think that rating is done with games played at depth 16 per move, if so then yes it wouldn't be very reliable, probably more of a quick check to see if anything gets broken.

Yes the graph on his website was what I was thinking of.

[Rebel]

Got a new champion, Sergio-2257, 64.1%, ~99 elo

Code: Select all

sergio-0109   63.6%   2020-07-29 01:09 | 5000 games
sergio-2214   62.9%   2020-07-29 22:15 | 5000 games
sergio-2138   63.4%   2020-07-28 23:44 | 5000 games
sergio-0111   62.4%   2020-07-31 01:11 | 5000 games
sergio-1209   63.4%   2020-08-01 12:21 | 5000 games
sergio-1515   62.3%   2020-08-01 19:35 | 5000 games
sergio-2257   64.1%   2020-08-02 23:13 | 5000 games

For a long time it had a score of 65% but in the last 1000 games it settled for 64.1%

Edit - will put this version (ready to use) on my website installed in Arena.

[JohnS]

Got a new champion, Sergio-2257, 64.1%, ~99 elo

Code: Select all

sergio-0109   63.6%   2020-07-29 01:09 | 5000 games
sergio-2214   62.9%   2020-07-29 22:15 | 5000 games
sergio-2138   63.4%   2020-07-28 23:44 | 5000 games
sergio-0111   62.4%   2020-07-31 01:11 | 5000 games
sergio-1209   63.4%   2020-08-01 12:21 | 5000 games
sergio-1515   62.3%   2020-08-01 19:35 | 5000 games
sergio-2257   64.1%   2020-08-02 23:13 | 5000 games

For a long time it had a score of 65% but in the last 1000 games it settled for 64.1%

Edit - will put this version (ready to use) on my website installed in Arena.

Wow that was quick The champs have a very short reign and who can keep up all the new challengers.

[MikeB]

Got a new champion, Sergio-2257, 64.1%, ~99 elo

Code: Select all

sergio-0109   63.6%   2020-07-29 01:09 | 5000 games
sergio-2214   62.9%   2020-07-29 22:15 | 5000 games
sergio-2138   63.4%   2020-07-28 23:44 | 5000 games
sergio-0111   62.4%   2020-07-31 01:11 | 5000 games
sergio-1209   63.4%   2020-08-01 12:21 | 5000 games
sergio-1515   62.3%   2020-08-01 19:35 | 5000 games
sergio-2257   64.1%   2020-08-02 23:13 | 5000 games

For a long time it had a score of 65% but in the last 1000 games it settled for 64.1%

Edit - will put this version (ready to use) on my website installed in Arena.

I also saw a nice gain with 2257:

Code: Select all

$ run
Settings ...

Current date : time (EDST)
Date: 08/02/20 : 03:09:48

CMD1=Stockfish-XI-NN.exe ENG1=St-0252 EVFILE1=./eval/20200731-0252.bin
CMD2=Stockfish-XI-NN.exe ENG2=St-1515 EVFILE2=./eval/20200801-1515.bin
CMD4=Black-Diamond-XI-NN.exe ENG4=BD-0252 EVFILE4=./eval/20200731-0252.bin

PGN File: c:/cluster.mfb/pgn/08020309.pgn
Games: 4000
Projected-> Time: 4h:22m:27s
50+1.00

Time Control-> base+inc: 50+1.00
Start running the chess match ...
/c/Users/MichaelB7/home/Github/cutechess/projects/cli/run: line 208: 2nd: command not found

#########################################################################################################
###                                              Summary                                              ###
#########################################################################################################

PGN File: c:/cluster.mfb/pgn/08020309.pgn
Time Control: Time Control-> base+inc: 50+1.00
Games: 4000
Threads: 1
Hash: 256

Current date : time (EDST)
Date: 08/02/20 : 09:25:28

Projected-> Time: 4h:22m:27s
     Run -> Time: 6h:15m:40s

6000 game(s) loaded
Rank Name     Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR
---------------------------------------------------------------------------------------------------------

   1 St-1515   3504   0.0    4    4  6000 3061.0  51.0  681  559 4760  11.3  79.3  3496
   2 St-0252   3496   7.0    4    4  6000 2939.0  49.0  559  681 4760   9.3  79.3  3504
---------------------------------------------------------------------------------------------------------

  Δ = delta from the next higher rated opponent
  # = number of games played
  Σ = total score, 1 point for win, 1/2 point for draw

LOS:
         St St
St-1515     99
St-0252   0

#########################################################################################################
###                                                End                                                ###
#########################################################################################################


6000 game(s) loaded

Speaking of 2257 < I have modified Honey to so you can run a bench with any net that you desired - as you may or may not know, every bin has a different bench signature
format is the normal bench with two additional parameters - one tha NN is true the name and location of file
Ho*r4*.exe b 16 1 13 true C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin >/dev/null

Code: Select all

Ho*r4*.exe  b 16 1 13 true C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin >/dev/null

Position: 1/45
Nodes/Second: 1182486
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 2/45
Nodes/Second: 1266516
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 3/45
Nodes/Second: 1813785
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 4/45
Nodes/Second: 1312303
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 5/45
Nodes/Second: 1170134
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 6/45
Nodes/Second: 1191088
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 7/45
Nodes/Second: 1220884
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 8/45
Nodes/Second: 1303099
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 9/45
Nodes/Second: 1215618
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 10/45
Nodes/Second: 1314588
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 11/45
Nodes/Second: 1231566
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 12/45
Nodes/Second: 1260754
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 13/45
Nodes/Second: 1325046
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 14/45
Nodes/Second: 1269061
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 15/45
Nodes/Second: 1414901
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 16/45
Nodes/Second: 1309956
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 17/45
Nodes/Second: 1429676
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 18/45
Nodes/Second: 1418933
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 19/45
Nodes/Second: 1848273
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 20/45
Nodes/Second: 1689525
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 21/45
Nodes/Second: 1968583
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 22/45
Nodes/Second: 2136333
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 23/45
Nodes/Second: 2351380
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 24/45
Nodes/Second: 1506656
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 25/45
Nodes/Second: 1413363
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 26/45
Nodes/Second: 1574833
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 27/45
Nodes/Second: 1466844
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 28/45
Nodes/Second: 1347375
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 29/45
Nodes/Second: 1395311
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 30/45
Nodes/Second: 1713045
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 31/45
Nodes/Second: 1425828
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 32/45
Nodes/Second: 1325442
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 33/45
Nodes/Second: 1246672
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 34/45
Nodes/Second: 1219804
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 35/45
Nodes/Second: 1581569
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 36/45
Nodes/Second: 1970937
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 37/45
Nodes/Second: 2260260
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 38/45
Nodes/Second: 2186288
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 39/45
Nodes/Second: 1901694
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 40/45
Nodes/Second: 1954192
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 41/45
Nodes/Second: 2242615
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 42/45
Nodes/Second: 1810852
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 43/45
Nodes/Second: 1121000
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 44/45
Nodes/Second: 1181090
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

Position: 45/45
Nodes/Second: 1147764
NN evaluation using C:/cluster.mfb/Popcnt-LP/eval/20200802-2257.bin enabled.

=================================
Total time (ms) : 3285
Nodes searched  : 4732771

Nodes/second    : 1440721

and lastly , you will be able to switch back and forth easily from the command line

right click and select open in new window to a more readable version

hit the return key to get current status"
"s nn true" to enable NN evaluation
run your search
"s nn false" enable classical evaluation
run your search
literally two engines at your disposal while running one exe

[peter]

and lastly , you will be able to switch back and forth easily from the command line

right click and select open in new window to a more readable version

hit the return key to get current status"
"s nn true" to enable NN evaluation
run your search
"s nn false" enable classical evaluation
run your search
literally two engines at your disposal while running one exe

Sounds great, Mike, will there be other compiles except the one you already made available?
I know, classical popcnt isn't really good for NNUE, but there are some kinds of "extra- popcnt" around. "Legacy", "ssse3-popcnt", "AIIAVXpopcnt" are some of the names, that did work with my old Xeon X5670.

[Milos]

and lastly , you will be able to switch back and forth easily from the command line

right click and select open in new window to a more readable version

hit the return key to get current status"
"s nn true" to enable NN evaluation
run your search
"s nn false" enable classical evaluation
run your search
literally two engines at your disposal while running one exe

Sounds great, Mike, will there be other compiles except the one you already made available?
I know, classical popcnt isn't really good for NNUE, but there are some kinds of "extra- popcnt" around. "Legacy", "ssse3-popcnt", "AIIAVXpopcnt" are some of the names, that did work with my old Xeon X5670.

The actual NNUE implementation is handcrafted and only has AVX2, SSE2 and SSE4.1 implementation. So no additional compiler switches would make any difference at all.