Sergio Vieri second net is out

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Re: Sergio Vieri second net is out

Post by MikeB »

MikeB wrote: Wed Jul 29, 2020 10:43 pm 2106 still on top, beating the net that Sergio called "best" 2138, quite convincingly with 95% los after 6000 games . 2214 was pretty close, surprisingly based on Sergio's own rating.

Code: Select all

$ run.sh
Settings ...

CMD1=Honey-XI-NN.exe ENG1=Ho-0629 EVFILE1=./eval/20200729-0629.bin
CMD2=Honey-XI-NN.exe ENG2=Ho-2138 EVFILE2=./eval/20200728-2138.bin
CMD3=Honey-XI-NN.exe ENG3=Ho-2214 EVFILE3=./eval/20200729-2214.bin

PGN File: c:/cluster.mfb/pgn/07291356.pgn
Games: 6000
Projected-> Time: 2h:30m:44s
20+0.40

Time Control-> base+inc: 20+0.40
Start running the chess match ...

#########################################################################################################
###                                              Summary                                              ###
#########################################################################################################

PGN File: c:/cluster.mfb/pgn/07291356.pgn
Time Control: Time Control-> base+inc: 20+0.40
Games: 6000
Threads: 1
Hash: 128

Current date : time (EDST)
Date: 07/29/20 : 16:35:27

Projected-> Time: 2h:30m:44s
     Run -> Time: 2h:38m:33s

6000 game(s) loaded
Rank Name     Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR
---------------------------------------------------------------------------------------------------------

   1 Ho-0629   3503   0.0    7    7  4000 2022.0  50.5  982  938 2080  24.6  52.0  3499
   2 Ho-2214   3502   0.5    7    7  4000 2017.0  50.4  951  917 2132  23.8  53.3  3499
   3 Ho-2138   3495   6.6    7    7  4000 1961.0  49.0  916  994 2090  22.9  52.2  3502
---------------------------------------------------------------------------------------------------------

  Δ = delta from the next higher rated opponent
  # = number of games played
  Σ = total score, 1 point for win, 1/2 point for draw

LOS:
         Ho Ho Ho
Ho-0629     54 95
Ho-2214  45    93
Ho-2138   4  6

#########################################################################################################
###                                                End                                                ###
#########################################################################################################
next up, regression tests 0629 vs the very first sergio net 1017 and cur-dev-stockfish, using a more normal opening file ( more draws)
pretty, pretty, good ...

Code: Select all

$ run.sh
Settings ...

CMD1=Honey-XI-NN.exe ENG1=Ho-0629 EVFILE1=./eval/20200729-0629.bin
CMD2=Honey-XI-NN.exe ENG2=Ho-1017 EVFILE2=./eval/20200720-1017.bin

PGN File: c:/cluster.mfb/pgn/07291643.pgn
Games: 3000
Projected-> Time: 1h:15m:32s
20+0.40

Time Control-> base+inc: 20+0.40

Start running the chess match ...

#########################################################################################################
###                                              Summary                                              ###
#########################################################################################################

PGN File: c:/cluster.mfb/pgn/07291643.pgn
Time Control: Time Control-> base+inc: 20+0.40
Games: 3000
Threads: 1
Hash: 128

Current date : time (EDST)
Date: 07/29/20 : 18:07:06

Projected-> Time: 1h:15m:32s
     Run -> Time: 1h:23m:17s

3000 game(s) loaded
Rank Name     Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR
---------------------------------------------------------------------------------------------------------

   1 Ho-0629   3551   0.0    9    9  2000 1224.5  61.2  615  166 1219  30.8  61.0  3474
   2 Ho-1017   3504  46.8    9    9  2000 1013.5  50.7  371  344 1285  18.6  64.2  3498
   3 sf        3444  60.2    9    9  2000  762.0  38.1  228  704 1068  11.4  53.4  3528
---------------------------------------------------------------------------------------------------------

  Δ = delta from the next higher rated opponent
  # = number of games played
  Σ = total score, 1 point for win, 1/2 point for draw

LOS:
         Ho Ho sf
Ho-0629    100100
Ho-1017   0   100
sf        0  0

#########################################################################################################
###                                                End                                                ###
#########################################################################################################

next set will be a a couple of quick sets vs cu-dev-sf to a depth of 5 with two different opening books
Image
User avatar
cdani
Posts: 2204
Joined: Sat Jan 18, 2014 10:24 am
Location: Andorra

Re: Sergio Vieri second net is out

Post by cdani »

Jouni wrote: Wed Jul 29, 2020 10:03 pm Only info is "d16" whatever it means?
Means that the graph is based on depth 16 tournament tests. Is a way of doing really fast tournaments, but is not very accurate. It's better games with time control. One net that is clearly better at the same depth can be worst on timed tournament.
User avatar
Ovyron
Posts: 4556
Joined: Tue Jul 03, 2007 4:30 am

Re: Sergio Vieri second net is out

Post by Ovyron »

cdani wrote: Thu Jul 30, 2020 12:28 am Means that the graph is based on depth 16 tournament tests. Is a way of doing really fast tournaments, but is not very accurate. It's better games with time control. One net that is clearly better at the same depth can be worst on timed tournament.
Yeah, but here all have same time management and same search, so fixed depth testing is sound (at least better than games with time control that have depth 15 or less per move in average).
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Re: Sergio Vieri second net is out

Post by MikeB »

both engines set to depth 5, a normal chess opening positions

Code: Select all

#########################################################################################################
###                                              Summary                                              ###
#########################################################################################################

PGN File: c:/cluster.mfb/pgn/07291912.pgn
Time Control: Time Control-> base+inc: 20+0.40
Games: 10000
Threads: 1
Hash: 128

Current date : time (EDST)
Date: 07/29/20 : 19:16:09

     Run -> Time: 0h:4m:3s

10000 game(s) loaded
Rank Name     Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR
---------------------------------------------------------------------------------------------------------

   1 Ho-0629   3569   0.0    7    7 10000 6875.0  68.8 6510 2760  730  65.1   7.3  3431
   2 sf        3431 138.1    7    7 10000 3125.0  31.2 2760 6510  730  27.6   7.3  3569
---------------------------------------------------------------------------------------------------------

  Δ = delta from the next higher rated opponent
  # = number of games played
  Σ = total score, 1 point for win, 1/2 point for draw

LOS:
         Ho sf
Ho-0629    100
sf        0

#########################################################################################################
###                                                End                                                ###
#########################################################################################################

same deal, but with NBSC_30k_5mvs.epd - this book is heavily unbalanced towards white - so in a search with a depth 5 , it probably helps the weaker engine when it is white

Code: Select all

 
#########################################################################################################
###                                              Summary                                              ###
#########################################################################################################

PGN File: c:/cluster.mfb/pgn/07291920.pgn
Time Control: Time Control-> base+inc: 20+0.40
Games: 10000
Threads: 1
Hash: 128

Current date : time (EDST)
Date: 07/29/20 : 19:24:21

     Run -> Time: 0h:4m:1s

10000 game(s) loaded
Rank Name     Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR
---------------------------------------------------------------------------------------------------------

   1 Ho-0629   3562   0.0    7    7 10000 6690.5  66.9 6257 2876  867  62.6   8.7  3438
   2 sf        3438 123.2    7    7 10000 3309.5  33.1 2876 6257  867  28.8   8.7  3562
---------------------------------------------------------------------------------------------------------

  Δ = delta from the next higher rated opponent
  # = number of games played
  Σ = total score, 1 point for win, 1/2 point for draw

LOS:
         Ho sf
Ho-0629    100
sf        0

#########################################################################################################
###                                                End                                                ###
#########################################################################################################
next up , Ho-0629 vs Komodo Depth 5...
Image
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Re: Sergio Vieri second net is out

Post by MikeB »

Komodo 14 vs Ho-0629 depth 5

Code: Select all

#########################################################################################################
###                                              Summary                                              ###
#########################################################################################################

PGN File: c:/cluster.mfb/pgn/07291936.pgn
Time Control: Time Control-> base+inc: 20+0.40
Games: 10000
Threads: 1
Hash: 128

Current date : time (EDST)
Date: 07/29/20 : 19:41:41

     Run -> Time: 0h:4m:49s

10000 game(s) loaded
Rank Name       Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR
---------------------------------------------------------------------------------------------------------

   1 komodo-14   3597   0.0    7    7 10000 7534.5  75.3 6953 1884 1163  69.5  11.6  3403
   2 Ho-0629     3403 194.8    7    7 10000 2465.5  24.7 1884 6953 1163  18.8  11.6  3597
---------------------------------------------------------------------------------------------------------

  Δ = delta from the next higher rated opponent
  # = number of games played
  Σ = total score, 1 point for win, 1/2 point for draw

LOS:
           ko Ho
komodo-14    100
Ho-0629     0

#########################################################################################################
###                                                End                                                ###
#########################################################################################################
next up Komodo 14 vs Ho-0629 50,000 nodes
Image
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Re: Sergio Vieri second net is out

Post by MikeB »

Komodo does not appear have a nodes=x game parameter - so here's go depth 15 results

Code: Select all

#########################################################################################################
###                                              Summary                                              ###
#########################################################################################################

PGN File: c:/cluster.mfb/pgn/07292049.pgn

Games: 1000
Threads: 1
Hash: 128

Current date : time (EDST)
Date: 07/29/20 : 20:52:56


     Run -> Time: 0h:3m:48s

1000 game(s) loaded
Rank Name       Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR
---------------------------------------------------------------------------------------------------------

   1 Ho-0629     3549   0.0   17   17  1000  637.5  63.7  461  186  353  46.1  35.3  3451
   2 komodo-14   3451  97.9   17   17  1000  362.5  36.2  186  461  353  18.6  35.3  3549
---------------------------------------------------------------------------------------------------------

  Δ = delta from the next higher rated opponent
  # = number of games played
  Σ = total score, 1 point for win, 1/2 point for draw

LOS:
           Ho ko
Ho-0629      100
komodo-14   0

#########################################################################################################
###                                                End                                                ###
#########################################################################################################
next up Ho-0629 v K, base+inc: 20+0.25 with K given 22+0.25 plus a two second overhead
Image
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Re: Sergio Vieri second net is out

Post by MikeB »

K was given TC:22+.25 with Overhead set at 2000ms to avoid time losses

Code: Select all

#########################################################################################################
###                                              Summary                                              ###
#########################################################################################################

PGN File: c:/cluster.mfb/pgn/07292104.pgn
Time Control: Time Control-> base+inc: 20+0.25
Games: 1000
Threads: 1
Hash: 128

Current date : time (EDST)
Date: 07/29/20 : 21:27:45

Projected-> Time: 0h:20m:4s
     Run -> Time: 0h:23m:32s

1000 game(s) loaded
Rank Name       Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR
---------------------------------------------------------------------------------------------------------

   1 Ho-0629     3610   0.0   13   13  1000  812.5  81.2  651   26  323  65.1  32.3  3390
   2 komodo-14   3390 219.4   13   13  1000  187.5  18.8   26  651  323   2.6  32.3  3610
---------------------------------------------------------------------------------------------------------

  Δ = delta from the next higher rated opponent
  # = number of games played
  Σ = total score, 1 point for win, 1/2 point for draw

LOS:
           Ho ko
Ho-0629      100
komodo-14   0

#########################################################################################################
###                                                End                                                ###
#########################################################################################################


1000 game(s) loaded
Image
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Re: Sergio Vieri second net is out

Post by MikeB »

an overnight run vs vs cu-dev-sf at longer tc, stopping it here

Code: Select all

PGN File: c:/cluster.mfb/pgn/07300054.pgn
Time Control: Time Control-> base+inc: 60+2.00
Games: 6000
Threads: 1
Hash: 256

Current date : time (EDST)
Date: 07/30/20 : 08:19:07

Projected-> Time: 10h:27m:32s
     Run -> Time: 7h:24m:12s

4104 game(s) loaded
Iteration 100: 9.44337e-05
Rank Name     Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR
---------------------------------------------------------------------------------------------------------

   1 Sf-0629   3524   0.0    6    6  2052 1135.5  55.3  326  107 1619  15.9  78.9  3492
   2 Bl-0629   3521   3.0    6    6  2052 1120.0  54.6  317  129 1606  15.4  78.3  3493
   3 Ho-0629   3521   0.2    6    6  2052 1119.5  54.6  330  143 1579  16.1  76.9  3493
   4 sf        3433  88.4    5    5  2052  729.0  35.5  117  711 1224   5.7  59.6  3522
---------------------------------------------------------------------------------------------------------
Image
Zenmastur
Posts: 919
Joined: Sat May 31, 2014 8:28 am

Re: Sergio Vieri second net is out

Post by Zenmastur »

I tested 2214 against the relatively early net 1134 both with PO-AVX2 @ 20+0.2 5000 games with a 4-move book as a sanity check for progress. The results were:

Code: Select all

 1.  SFSF-NNUE-2214           	2645.0/5000	922-632-3446  	
 2.  SFSF-NNUE-1134           	2355.0/5000	632-922-3446  	
About a 20 ELO difference with 7 ELO error bars if my guess is correct.

Regards,

Zenmastur
Only 2 defining forces have ever offered to die for you.....Jesus Christ and the American Soldier. One died for your soul, the other for your freedom.
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Re: Sergio Vieri second net is out

Post by MikeB »

a virtual tie , but ties go the current champ, 0629, went a little longer here to see if an engine would breakout - but none did - just about dead even - next up 2333
tc:20+0.40

Code: Select all

Rank Name     Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR
---------------------------------------------------------------------------------------------------------
   1 Ho-0629   3100   0.0    5    5  8000 4003.5  50.0 1571 1564 4865  19.6  60.8  3100
   2 Ho-1934   3100   0.2    5    5  8000 3996.5  50.0 1564 1571 4865  19.6  60.8  3100
---------------------------------------------------------------------------------------------------------
  Δ = delta from the next higher rated opponent
  # = number of games played
  Σ = total score, 1 point for win, 1/2 point for draw

ResultSet-EloRating>los
         Ho Ho
Ho-0629     53
Ho-1934  46
ResultSet-EloRating>
Image