Can the sardine! NNUE clobbers SF.

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: Can the sardine! NNUE clobbers SF.

Post by Laskos »

Rebel wrote: Fri Jul 17, 2020 1:20 pm 2000 40m/20s games

Code: Select all

Score of henk vs sf11: 621 - 433 - 946  [0.547] 2000
...      henk playing White: 381 - 150 - 469  [0.616] 1000
...      henk playing Black: 240 - 283 - 477  [0.478] 1000
...      White vs Black: 664 - 390 - 946  [0.569] 2000
Elo difference: 32.8 +/- 11.0, LOS: 100.0 %, DrawRatio: 47.3 %
Finished match

Performs better than SF_dev against Lc0 GPU engines + nets too. Seems to be the strongest chess entity on many balanced CPU/GPU systems. Pity I cannot run Sim with it, I would be curious to see similarities (with Lc0 GPU and SF_dev).
Ferdy
Posts: 4840
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Can the sardine! NNUE clobbers SF.

Post by Ferdy »

Rebel wrote: Fri Jul 17, 2020 10:24 amThe similarity with the Lc0 related nets would be interesting.
Run sfnn, sf and lc0 blas and added from my old list. Stockfish 2020-07-11 is a dev.

Image

Matrix, engines are run with different movetime. sf10 and sf10 r1 is the same as ref.

Code: Select all

sim version 3

  Key:

  1) Amoeba 2.8 (time: 200 ms scale: 1.0)
  2) Andscacs 0.95 (time: 200 ms scale: 1.0)
  3) Arasan 21.1 (time: 200 ms scale: 1.0)
  4) Atlas 3.91 (time: 200 ms scale: 1.0)
  5) Bobcat v8.0 (time: 200 ms scale: 1.0)
  6) Booot 6.3.1 (time: 200 ms scale: 1.0)
  7) Cheng 4.39 (time: 200 ms scale: 1.0)
  8) Demolito 2018-10-29 (time: 200 ms scale: 1.0)
  9) Ethereal 11.25 (time: 200 ms scale: 1.0)
 10) Fizbo 2 (time: 200 ms scale: 1.0)
 11) Fruit 2.1 (time: 200 ms scale: 1.0)
 12) Fruit reloaded 3.2.1 x64-pop (time: 200 ms scale: 1.0)
 13) GreKo 2018.08 (time: 200 ms scale: 1.0)
 14) Gull 3 x64 (time: 200 ms scale: 1.0)
 15) Hannibal 1.7 x64 (time: 200 ms scale: 1.0)
 16) iCE 3.0 v658 (time: 200 ms scale: 1.0)
 17) Laser 1.7 (time: 200 ms scale: 1.0)
 18) Lc0 v0.26.0 blas w591226 (time: 200 ms scale: 1.0)
 19) Nemorino 5.00 (time: 200 ms scale: 1.0)
 20) Pedone 1.9 (time: 200 ms scale: 1.0)
 21) Rodent 0.275 (time: 200 ms scale: 1.0)
 22) RofChade Version 2.0 (time: 200 ms scale: 1.0)
 23) Senpai 2.0 (time: 200 ms scale: 1.0)
 24) SFNNUE_2020-07-15_GK_06-27 (time: 10 ms scale: 1.0)
 25) SmarThink 1.98 (time: 200 ms scale: 1.0)
 26) Stockfish 10 (time: 20 ms scale: 1.0)
 27) Stockfish 10 r1 (time: 20 ms scale: 1.0)
 28) Stockfish 11 (time: 10 ms scale: 1.0)
 29) Stockfish 2020-07-11 (time: 10 ms scale: 1.0)
 30) Texel 1.07 (time: 200 ms scale: 1.0)
 31) The Baron 3.44 (time: 200 ms scale: 1.0)
 32) Toga II 4.01 (time: 200 ms scale: 1.0)
 33) Vajolet2 2.6.1 (time: 200 ms scale: 1.0)
 34) Wasp 3.50 (time: 200 ms scale: 1.0)
 35) Winter 0.5 (time: 200 ms scale: 1.0)
 36) Xiphos 0.5 (time: 200 ms scale: 1.0)
 37) zurichess neuchatel (time: 200 ms scale: 1.0)

         1     2     3     4     5     6     7     8     9    10    11    12    13    14    15    16    17    18    19    20    21    22    23    24    25    26    27    28    29    30    31    32    33    34    35    36    37
  1.  ----- 20.32 25.20 25.69 27.14 22.55 19.66 25.46 18.17 17.36 27.01 10.18 27.09 24.41 23.49 22.10 16.64 12.95 14.65 21.61 26.33 23.04 25.53 17.47 14.36 19.86 20.02 16.16 19.06 24.87 22.99 26.34 18.38 23.55 24.76 15.99 26.51
  2.  20.32 ----- 40.16 40.56 40.62 40.40 29.70 42.69 32.82 29.16 35.34 11.12 37.81 43.53 40.18 36.60 28.74 17.32 21.09 31.71 41.19 41.15 43.03 23.59 20.73 33.41 33.99 21.74 26.63 41.85 32.90 40.03 29.97 40.08 36.05 26.95 38.15
  3.  25.20 40.16 ----- 45.28 48.24 41.56 33.53 46.72 34.07 31.03 43.11 12.39 44.34 46.65 42.72 40.43 29.23 18.03 23.27 35.37 47.49 44.72 47.73 26.73 22.49 36.70 36.96 23.85 30.31 47.45 38.91 44.99 32.73 44.77 40.69 27.74 45.14
  4.  25.69 40.56 45.28 ----- 47.48 42.00 33.13 47.97 32.41 29.36 47.04 13.09 44.84 45.85 45.36 38.99 28.70 18.66 24.00 37.56 46.55 43.89 46.83 25.47 23.59 34.58 35.18 23.10 29.25 45.51 38.63 48.62 33.09 45.57 42.30 27.54 45.71
  5.  27.14 40.62 48.24 47.48 ----- 41.92 36.57 48.49 34.83 31.39 47.62 12.82 48.66 47.57 43.26 42.77 30.23 18.63 23.80 37.48 50.90 46.84 49.88 25.97 22.35 35.63 36.51 24.07 30.42 50.30 41.64 48.66 32.07 44.91 44.06 28.73 49.48
  6.  22.55 40.40 41.56 42.00 41.92 ----- 30.26 45.52 33.52 29.63 39.01 12.07 39.79 45.24 41.67 38.03 29.51 17.52 23.08 33.47 43.26 42.51 43.28 24.65 20.48 35.26 36.25 22.92 28.50 42.52 34.40 43.81 31.74 43.28 37.22 26.73 41.48
  7.  19.66 29.70 33.53 33.13 36.57 30.26 ----- 32.92 24.62 23.71 31.86 11.34 32.30 33.66 30.83 30.72 22.14 14.82 19.12 26.68 34.74 31.97 34.12 18.85 17.54 25.49 26.27 17.29 21.19 34.45 28.91 33.54 23.96 32.20 30.70 21.57 32.48
  8.  25.46 42.69 46.72 47.97 48.49 45.52 32.92 ----- 35.13 31.46 44.80 12.75 46.64 48.92 45.55 42.26 30.49 19.24 25.42 38.77 46.52 47.35 49.14 26.38 23.27 37.63 38.50 23.55 30.76 46.30 39.33 48.37 34.62 47.45 42.67 29.40 48.73
  9.  18.17 32.82 34.07 32.41 34.83 33.52 24.62 35.13 ----- 24.97 28.44 10.34 31.57 35.65 32.06 31.29 26.40 15.77 19.59 26.88 33.93 36.96 36.21 20.38 17.37 29.15 29.40 18.28 22.61 35.34 28.07 32.22 25.65 33.76 30.23 22.98 32.61
 10.  17.36 29.16 31.03 29.36 31.39 29.63 23.71 31.46 24.97 ----- 26.78  9.53 29.59 31.35 29.36 28.94 22.29 16.19 17.35 24.99 31.83 31.31 32.47 21.36 17.43 26.26 26.81 17.47 21.92 33.02 26.60 28.84 22.83 29.59 27.94 21.50 30.27
 11.  27.01 35.34 43.11 47.04 47.62 39.01 31.86 44.80 28.44 26.78 ----- 16.27 48.11 43.37 40.87 36.19 26.17 17.07 23.37 35.47 46.75 39.60 43.01 25.26 21.83 32.37 32.90 23.16 28.93 43.12 39.28 55.23 29.83 43.12 41.99 24.47 45.01
 12.  10.18 11.12 12.39 13.09 12.82 12.07 11.34 12.75 10.34  9.53 16.27 ----- 13.10 11.85 11.93 11.68 10.63  8.38 10.43 11.51 13.04 11.52 12.09  8.59  8.25 10.06 10.46  8.82  9.40 12.01 12.67 15.09 11.76 13.06 12.41 10.21 12.45
 13.  27.09 37.81 44.34 44.84 48.66 39.79 32.30 46.64 31.57 29.59 48.11 13.10 ----- 43.43 40.79 38.44 28.14 18.15 23.04 35.75 46.95 42.44 47.50 25.35 20.87 34.06 34.60 22.94 29.23 44.09 40.45 46.97 31.59 42.77 43.71 26.22 46.13
 14.  24.41 43.53 46.65 45.85 47.57 45.24 33.66 48.92 35.65 31.35 43.37 11.85 43.43 ----- 44.45 41.85 31.31 18.41 24.13 36.36 47.48 46.78 49.24 25.53 23.02 36.94 37.22 23.85 29.41 47.24 37.93 46.50 34.07 45.87 40.19 29.01 45.63
 15.  23.49 40.18 42.72 45.36 43.26 41.67 30.83 45.55 32.06 29.36 40.87 11.93 40.79 44.45 ----- 36.97 28.82 17.75 23.84 34.32 43.81 41.85 44.76 24.16 21.49 34.38 35.08 22.23 28.48 42.57 37.00 44.55 32.14 43.18 39.20 26.94 41.99
 16.  22.10 36.60 40.43 38.99 42.77 38.03 30.72 42.26 31.29 28.94 36.19 11.68 38.44 41.85 36.97 ----- 26.91 16.91 21.80 31.35 41.21 41.20 42.38 23.39 20.47 31.56 32.54 21.15 25.26 41.62 33.95 39.35 29.16 38.70 35.76 26.68 40.31
 17.  16.64 28.74 29.23 28.70 30.23 29.51 22.14 30.49 26.40 22.29 26.17 10.63 28.14 31.31 28.82 26.91 ----- 13.89 17.39 25.14 30.10 30.66 31.20 17.12 14.76 25.69 25.78 16.40 19.85 30.71 24.42 28.96 23.05 30.35 26.88 20.61 28.21
 18.  12.95 17.32 18.03 18.66 18.63 17.52 14.82 19.24 15.77 16.19 17.07  8.38 18.15 18.41 17.75 16.91 13.89 ----- 11.88 15.34 18.57 18.86 18.83 15.49 10.59 16.87 17.50 12.94 15.17 18.84 16.17 17.82 14.51 18.16 17.61 13.63 18.09
 19.  14.65 21.09 23.27 24.00 23.80 23.08 19.12 25.42 19.59 17.35 23.37 10.43 23.04 24.13 23.84 21.80 17.39 11.88 ----- 20.49 24.19 22.93 25.01 14.53 13.18 20.45 21.36 14.82 16.52 23.82 20.98 24.68 20.66 24.70 22.06 17.32 23.32
 20.  21.61 31.71 35.37 37.56 37.48 33.47 26.68 38.77 26.88 24.99 35.47 11.51 35.75 36.36 34.32 31.35 25.14 15.34 20.49 ----- 36.88 35.52 37.11 22.09 17.88 29.93 30.81 20.65 25.39 36.77 30.77 36.03 27.77 35.65 34.18 24.12 37.27
 21.  26.33 41.19 47.49 46.55 50.90 43.26 34.74 46.52 33.93 31.83 46.75 13.04 46.95 47.48 43.81 41.21 30.10 18.57 24.19 36.88 ----- 46.44 48.39 26.77 22.58 36.60 37.62 24.25 31.50 49.15 40.33 48.79 32.76 45.82 43.25 28.85 48.92
 22.  23.04 41.15 44.72 43.89 46.84 42.51 31.97 47.35 36.96 31.31 39.60 11.52 42.44 46.78 41.85 41.20 30.66 18.86 22.93 35.52 46.44 ----- 46.49 25.42 21.63 37.11 37.96 23.66 30.15 46.32 35.89 43.21 32.35 43.70 40.20 29.61 45.05
 23.  25.53 43.03 47.73 46.83 49.88 43.28 34.12 49.14 36.21 32.47 43.01 12.09 47.50 49.24 44.76 42.38 31.20 18.83 25.01 37.11 48.39 46.49 ----- 27.18 22.88 38.06 38.75 24.06 31.09 48.59 39.99 46.98 33.42 46.41 43.35 29.97 47.82
 24.  17.47 23.59 26.73 25.47 25.97 24.65 18.85 26.38 20.38 21.36 25.26  8.59 25.35 25.53 24.16 23.39 17.12 15.49 14.53 22.09 26.77 25.42 27.18 ----- 15.19 25.09 25.38 20.82 24.59 26.18 23.34 25.35 18.89 24.42 24.45 17.70 26.66
 25.  14.36 20.73 22.49 23.59 22.35 20.48 17.54 23.27 17.37 17.43 21.83  8.25 20.87 23.02 21.49 20.47 14.76 10.59 13.18 17.88 22.58 21.63 22.88 15.19 ----- 18.06 18.39 13.57 16.12 22.12 20.53 22.60 16.81 21.91 19.82 15.22 22.41
 26.  19.86 33.41 36.70 34.58 35.63 35.26 25.49 37.63 29.15 26.26 32.37 10.06 34.06 36.94 34.38 31.56 25.69 16.87 20.45 29.93 36.60 37.11 38.06 25.09 18.06 ----- 68.67 24.02 30.13 37.10 31.03 34.22 26.40 35.28 32.50 24.63 36.13
 27.  20.02 33.99 36.96 35.18 36.51 36.25 26.27 38.50 29.40 26.81 32.90 10.46 34.60 37.22 35.08 32.54 25.78 17.50 21.36 30.81 37.62 37.96 38.75 25.38 18.39 68.67 ----- 24.70 30.95 37.79 31.62 35.54 27.23 36.03 33.78 24.82 36.06
 28.  16.16 21.74 23.85 23.10 24.07 22.92 17.29 23.55 18.28 17.47 23.16  8.82 22.94 23.85 22.23 21.15 16.40 12.94 14.82 20.65 24.25 23.66 24.06 20.82 13.57 24.02 24.70 ----- 28.07 23.25 21.47 22.63 17.97 23.09 22.87 16.50 24.17
 29.  19.06 26.63 30.31 29.25 30.42 28.50 21.19 30.76 22.61 21.92 28.93  9.40 29.23 29.41 28.48 25.26 19.85 15.17 16.52 25.39 31.50 30.15 31.09 24.59 16.12 30.13 30.95 28.07 ----- 30.57 26.40 29.22 21.12 28.51 28.53 20.16 30.57
 30.  24.87 41.85 47.45 45.51 50.30 42.52 34.45 46.30 35.34 33.02 43.12 12.01 44.09 47.24 42.57 41.62 30.71 18.84 23.82 36.77 49.15 46.32 48.59 26.18 22.12 37.10 37.79 23.25 30.57 ----- 38.65 46.04 31.88 45.74 42.34 29.11 46.21
 31.  22.99 32.90 38.91 38.63 41.64 34.40 28.91 39.33 28.07 26.60 39.28 12.67 40.45 37.93 37.00 33.95 24.42 16.17 20.98 30.77 40.33 35.89 39.99 23.34 20.53 31.03 31.62 21.47 26.40 38.65 ----- 39.33 27.29 37.11 36.84 23.80 39.58
 32.  26.34 40.03 44.99 48.62 48.66 43.81 33.54 48.37 32.22 28.84 55.23 15.09 46.97 46.50 44.55 39.35 28.96 17.82 24.68 36.03 48.79 43.21 46.98 25.35 22.60 34.22 35.54 22.63 29.22 46.04 39.33 ----- 32.64 46.49 43.02 25.94 46.37
 33.  18.38 29.97 32.73 33.09 32.07 31.74 23.96 34.62 25.65 22.83 29.83 11.76 31.59 34.07 32.14 29.16 23.05 14.51 20.66 27.77 32.76 32.35 33.42 18.89 16.81 26.40 27.23 17.97 21.12 31.88 27.29 32.64 ----- 32.51 29.11 21.74 31.23
 34.  23.55 40.08 44.77 45.57 44.91 43.28 32.20 47.45 33.76 29.59 43.12 13.06 42.77 45.87 43.18 38.70 30.35 18.16 24.70 35.65 45.82 43.70 46.41 24.42 21.91 35.28 36.03 23.09 28.51 45.74 37.11 46.49 32.51 ----- 39.33 26.56 43.18
 35.  24.76 36.05 40.69 42.30 44.06 37.22 30.70 42.67 30.23 27.94 41.99 12.41 43.71 40.19 39.20 35.76 26.88 17.61 22.06 34.18 43.25 40.20 43.35 24.45 19.82 32.50 33.78 22.87 28.53 42.34 36.84 43.02 29.11 39.33 ----- 25.53 43.11
 36.  15.99 26.95 27.74 27.54 28.73 26.73 21.57 29.40 22.98 21.50 24.47 10.21 26.22 29.01 26.94 26.68 20.61 13.63 17.32 24.12 28.85 29.61 29.97 17.70 15.22 24.63 24.82 16.50 20.16 29.11 23.80 25.94 21.74 26.56 25.53 ----- 28.71
 37.  26.51 38.15 45.14 45.71 49.48 41.48 32.48 48.73 32.61 30.27 45.01 12.45 46.13 45.63 41.99 40.31 28.21 18.09 23.32 37.27 48.92 45.05 47.82 26.66 22.41 36.13 36.06 24.17 30.57 46.21 39.58 46.37 31.23 43.18 43.11 28.71 -----


User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: Can the sardine! NNUE clobbers SF.

Post by Laskos »

Ferdy wrote: Sat Jul 18, 2020 10:27 am
Rebel wrote: Fri Jul 17, 2020 10:24 amThe similarity with the Lc0 related nets would be interesting.
Run sfnn, sf and lc0 blas and added from my old list. Stockfish 2020-07-11 is a dev.

Image

Matrix, engines are run with different movetime. sf10 and sf10 r1 is the same as ref.

Code: Select all

sim version 3

  Key:

  1) Amoeba 2.8 (time: 200 ms scale: 1.0)
  2) Andscacs 0.95 (time: 200 ms scale: 1.0)
  3) Arasan 21.1 (time: 200 ms scale: 1.0)
  4) Atlas 3.91 (time: 200 ms scale: 1.0)
  5) Bobcat v8.0 (time: 200 ms scale: 1.0)
  6) Booot 6.3.1 (time: 200 ms scale: 1.0)
  7) Cheng 4.39 (time: 200 ms scale: 1.0)
  8) Demolito 2018-10-29 (time: 200 ms scale: 1.0)
  9) Ethereal 11.25 (time: 200 ms scale: 1.0)
 10) Fizbo 2 (time: 200 ms scale: 1.0)
 11) Fruit 2.1 (time: 200 ms scale: 1.0)
 12) Fruit reloaded 3.2.1 x64-pop (time: 200 ms scale: 1.0)
 13) GreKo 2018.08 (time: 200 ms scale: 1.0)
 14) Gull 3 x64 (time: 200 ms scale: 1.0)
 15) Hannibal 1.7 x64 (time: 200 ms scale: 1.0)
 16) iCE 3.0 v658 (time: 200 ms scale: 1.0)
 17) Laser 1.7 (time: 200 ms scale: 1.0)
 18) Lc0 v0.26.0 blas w591226 (time: 200 ms scale: 1.0)
 19) Nemorino 5.00 (time: 200 ms scale: 1.0)
 20) Pedone 1.9 (time: 200 ms scale: 1.0)
 21) Rodent 0.275 (time: 200 ms scale: 1.0)
 22) RofChade Version 2.0 (time: 200 ms scale: 1.0)
 23) Senpai 2.0 (time: 200 ms scale: 1.0)
 24) SFNNUE_2020-07-15_GK_06-27 (time: 10 ms scale: 1.0)
 25) SmarThink 1.98 (time: 200 ms scale: 1.0)
 26) Stockfish 10 (time: 20 ms scale: 1.0)
 27) Stockfish 10 r1 (time: 20 ms scale: 1.0)
 28) Stockfish 11 (time: 10 ms scale: 1.0)
 29) Stockfish 2020-07-11 (time: 10 ms scale: 1.0)
 30) Texel 1.07 (time: 200 ms scale: 1.0)
 31) The Baron 3.44 (time: 200 ms scale: 1.0)
 32) Toga II 4.01 (time: 200 ms scale: 1.0)
 33) Vajolet2 2.6.1 (time: 200 ms scale: 1.0)
 34) Wasp 3.50 (time: 200 ms scale: 1.0)
 35) Winter 0.5 (time: 200 ms scale: 1.0)
 36) Xiphos 0.5 (time: 200 ms scale: 1.0)
 37) zurichess neuchatel (time: 200 ms scale: 1.0)

         1     2     3     4     5     6     7     8     9    10    11    12    13    14    15    16    17    18    19    20    21    22    23    24    25    26    27    28    29    30    31    32    33    34    35    36    37
  1.  ----- 20.32 25.20 25.69 27.14 22.55 19.66 25.46 18.17 17.36 27.01 10.18 27.09 24.41 23.49 22.10 16.64 12.95 14.65 21.61 26.33 23.04 25.53 17.47 14.36 19.86 20.02 16.16 19.06 24.87 22.99 26.34 18.38 23.55 24.76 15.99 26.51
  2.  20.32 ----- 40.16 40.56 40.62 40.40 29.70 42.69 32.82 29.16 35.34 11.12 37.81 43.53 40.18 36.60 28.74 17.32 21.09 31.71 41.19 41.15 43.03 23.59 20.73 33.41 33.99 21.74 26.63 41.85 32.90 40.03 29.97 40.08 36.05 26.95 38.15
  3.  25.20 40.16 ----- 45.28 48.24 41.56 33.53 46.72 34.07 31.03 43.11 12.39 44.34 46.65 42.72 40.43 29.23 18.03 23.27 35.37 47.49 44.72 47.73 26.73 22.49 36.70 36.96 23.85 30.31 47.45 38.91 44.99 32.73 44.77 40.69 27.74 45.14
  4.  25.69 40.56 45.28 ----- 47.48 42.00 33.13 47.97 32.41 29.36 47.04 13.09 44.84 45.85 45.36 38.99 28.70 18.66 24.00 37.56 46.55 43.89 46.83 25.47 23.59 34.58 35.18 23.10 29.25 45.51 38.63 48.62 33.09 45.57 42.30 27.54 45.71
  5.  27.14 40.62 48.24 47.48 ----- 41.92 36.57 48.49 34.83 31.39 47.62 12.82 48.66 47.57 43.26 42.77 30.23 18.63 23.80 37.48 50.90 46.84 49.88 25.97 22.35 35.63 36.51 24.07 30.42 50.30 41.64 48.66 32.07 44.91 44.06 28.73 49.48
  6.  22.55 40.40 41.56 42.00 41.92 ----- 30.26 45.52 33.52 29.63 39.01 12.07 39.79 45.24 41.67 38.03 29.51 17.52 23.08 33.47 43.26 42.51 43.28 24.65 20.48 35.26 36.25 22.92 28.50 42.52 34.40 43.81 31.74 43.28 37.22 26.73 41.48
  7.  19.66 29.70 33.53 33.13 36.57 30.26 ----- 32.92 24.62 23.71 31.86 11.34 32.30 33.66 30.83 30.72 22.14 14.82 19.12 26.68 34.74 31.97 34.12 18.85 17.54 25.49 26.27 17.29 21.19 34.45 28.91 33.54 23.96 32.20 30.70 21.57 32.48
  8.  25.46 42.69 46.72 47.97 48.49 45.52 32.92 ----- 35.13 31.46 44.80 12.75 46.64 48.92 45.55 42.26 30.49 19.24 25.42 38.77 46.52 47.35 49.14 26.38 23.27 37.63 38.50 23.55 30.76 46.30 39.33 48.37 34.62 47.45 42.67 29.40 48.73
  9.  18.17 32.82 34.07 32.41 34.83 33.52 24.62 35.13 ----- 24.97 28.44 10.34 31.57 35.65 32.06 31.29 26.40 15.77 19.59 26.88 33.93 36.96 36.21 20.38 17.37 29.15 29.40 18.28 22.61 35.34 28.07 32.22 25.65 33.76 30.23 22.98 32.61
 10.  17.36 29.16 31.03 29.36 31.39 29.63 23.71 31.46 24.97 ----- 26.78  9.53 29.59 31.35 29.36 28.94 22.29 16.19 17.35 24.99 31.83 31.31 32.47 21.36 17.43 26.26 26.81 17.47 21.92 33.02 26.60 28.84 22.83 29.59 27.94 21.50 30.27
 11.  27.01 35.34 43.11 47.04 47.62 39.01 31.86 44.80 28.44 26.78 ----- 16.27 48.11 43.37 40.87 36.19 26.17 17.07 23.37 35.47 46.75 39.60 43.01 25.26 21.83 32.37 32.90 23.16 28.93 43.12 39.28 55.23 29.83 43.12 41.99 24.47 45.01
 12.  10.18 11.12 12.39 13.09 12.82 12.07 11.34 12.75 10.34  9.53 16.27 ----- 13.10 11.85 11.93 11.68 10.63  8.38 10.43 11.51 13.04 11.52 12.09  8.59  8.25 10.06 10.46  8.82  9.40 12.01 12.67 15.09 11.76 13.06 12.41 10.21 12.45
 13.  27.09 37.81 44.34 44.84 48.66 39.79 32.30 46.64 31.57 29.59 48.11 13.10 ----- 43.43 40.79 38.44 28.14 18.15 23.04 35.75 46.95 42.44 47.50 25.35 20.87 34.06 34.60 22.94 29.23 44.09 40.45 46.97 31.59 42.77 43.71 26.22 46.13
 14.  24.41 43.53 46.65 45.85 47.57 45.24 33.66 48.92 35.65 31.35 43.37 11.85 43.43 ----- 44.45 41.85 31.31 18.41 24.13 36.36 47.48 46.78 49.24 25.53 23.02 36.94 37.22 23.85 29.41 47.24 37.93 46.50 34.07 45.87 40.19 29.01 45.63
 15.  23.49 40.18 42.72 45.36 43.26 41.67 30.83 45.55 32.06 29.36 40.87 11.93 40.79 44.45 ----- 36.97 28.82 17.75 23.84 34.32 43.81 41.85 44.76 24.16 21.49 34.38 35.08 22.23 28.48 42.57 37.00 44.55 32.14 43.18 39.20 26.94 41.99
 16.  22.10 36.60 40.43 38.99 42.77 38.03 30.72 42.26 31.29 28.94 36.19 11.68 38.44 41.85 36.97 ----- 26.91 16.91 21.80 31.35 41.21 41.20 42.38 23.39 20.47 31.56 32.54 21.15 25.26 41.62 33.95 39.35 29.16 38.70 35.76 26.68 40.31
 17.  16.64 28.74 29.23 28.70 30.23 29.51 22.14 30.49 26.40 22.29 26.17 10.63 28.14 31.31 28.82 26.91 ----- 13.89 17.39 25.14 30.10 30.66 31.20 17.12 14.76 25.69 25.78 16.40 19.85 30.71 24.42 28.96 23.05 30.35 26.88 20.61 28.21
 18.  12.95 17.32 18.03 18.66 18.63 17.52 14.82 19.24 15.77 16.19 17.07  8.38 18.15 18.41 17.75 16.91 13.89 ----- 11.88 15.34 18.57 18.86 18.83 15.49 10.59 16.87 17.50 12.94 15.17 18.84 16.17 17.82 14.51 18.16 17.61 13.63 18.09
 19.  14.65 21.09 23.27 24.00 23.80 23.08 19.12 25.42 19.59 17.35 23.37 10.43 23.04 24.13 23.84 21.80 17.39 11.88 ----- 20.49 24.19 22.93 25.01 14.53 13.18 20.45 21.36 14.82 16.52 23.82 20.98 24.68 20.66 24.70 22.06 17.32 23.32
 20.  21.61 31.71 35.37 37.56 37.48 33.47 26.68 38.77 26.88 24.99 35.47 11.51 35.75 36.36 34.32 31.35 25.14 15.34 20.49 ----- 36.88 35.52 37.11 22.09 17.88 29.93 30.81 20.65 25.39 36.77 30.77 36.03 27.77 35.65 34.18 24.12 37.27
 21.  26.33 41.19 47.49 46.55 50.90 43.26 34.74 46.52 33.93 31.83 46.75 13.04 46.95 47.48 43.81 41.21 30.10 18.57 24.19 36.88 ----- 46.44 48.39 26.77 22.58 36.60 37.62 24.25 31.50 49.15 40.33 48.79 32.76 45.82 43.25 28.85 48.92
 22.  23.04 41.15 44.72 43.89 46.84 42.51 31.97 47.35 36.96 31.31 39.60 11.52 42.44 46.78 41.85 41.20 30.66 18.86 22.93 35.52 46.44 ----- 46.49 25.42 21.63 37.11 37.96 23.66 30.15 46.32 35.89 43.21 32.35 43.70 40.20 29.61 45.05
 23.  25.53 43.03 47.73 46.83 49.88 43.28 34.12 49.14 36.21 32.47 43.01 12.09 47.50 49.24 44.76 42.38 31.20 18.83 25.01 37.11 48.39 46.49 ----- 27.18 22.88 38.06 38.75 24.06 31.09 48.59 39.99 46.98 33.42 46.41 43.35 29.97 47.82
 24.  17.47 23.59 26.73 25.47 25.97 24.65 18.85 26.38 20.38 21.36 25.26  8.59 25.35 25.53 24.16 23.39 17.12 15.49 14.53 22.09 26.77 25.42 27.18 ----- 15.19 25.09 25.38 20.82 24.59 26.18 23.34 25.35 18.89 24.42 24.45 17.70 26.66
 25.  14.36 20.73 22.49 23.59 22.35 20.48 17.54 23.27 17.37 17.43 21.83  8.25 20.87 23.02 21.49 20.47 14.76 10.59 13.18 17.88 22.58 21.63 22.88 15.19 ----- 18.06 18.39 13.57 16.12 22.12 20.53 22.60 16.81 21.91 19.82 15.22 22.41
 26.  19.86 33.41 36.70 34.58 35.63 35.26 25.49 37.63 29.15 26.26 32.37 10.06 34.06 36.94 34.38 31.56 25.69 16.87 20.45 29.93 36.60 37.11 38.06 25.09 18.06 ----- 68.67 24.02 30.13 37.10 31.03 34.22 26.40 35.28 32.50 24.63 36.13
 27.  20.02 33.99 36.96 35.18 36.51 36.25 26.27 38.50 29.40 26.81 32.90 10.46 34.60 37.22 35.08 32.54 25.78 17.50 21.36 30.81 37.62 37.96 38.75 25.38 18.39 68.67 ----- 24.70 30.95 37.79 31.62 35.54 27.23 36.03 33.78 24.82 36.06
 28.  16.16 21.74 23.85 23.10 24.07 22.92 17.29 23.55 18.28 17.47 23.16  8.82 22.94 23.85 22.23 21.15 16.40 12.94 14.82 20.65 24.25 23.66 24.06 20.82 13.57 24.02 24.70 ----- 28.07 23.25 21.47 22.63 17.97 23.09 22.87 16.50 24.17
 29.  19.06 26.63 30.31 29.25 30.42 28.50 21.19 30.76 22.61 21.92 28.93  9.40 29.23 29.41 28.48 25.26 19.85 15.17 16.52 25.39 31.50 30.15 31.09 24.59 16.12 30.13 30.95 28.07 ----- 30.57 26.40 29.22 21.12 28.51 28.53 20.16 30.57
 30.  24.87 41.85 47.45 45.51 50.30 42.52 34.45 46.30 35.34 33.02 43.12 12.01 44.09 47.24 42.57 41.62 30.71 18.84 23.82 36.77 49.15 46.32 48.59 26.18 22.12 37.10 37.79 23.25 30.57 ----- 38.65 46.04 31.88 45.74 42.34 29.11 46.21
 31.  22.99 32.90 38.91 38.63 41.64 34.40 28.91 39.33 28.07 26.60 39.28 12.67 40.45 37.93 37.00 33.95 24.42 16.17 20.98 30.77 40.33 35.89 39.99 23.34 20.53 31.03 31.62 21.47 26.40 38.65 ----- 39.33 27.29 37.11 36.84 23.80 39.58
 32.  26.34 40.03 44.99 48.62 48.66 43.81 33.54 48.37 32.22 28.84 55.23 15.09 46.97 46.50 44.55 39.35 28.96 17.82 24.68 36.03 48.79 43.21 46.98 25.35 22.60 34.22 35.54 22.63 29.22 46.04 39.33 ----- 32.64 46.49 43.02 25.94 46.37
 33.  18.38 29.97 32.73 33.09 32.07 31.74 23.96 34.62 25.65 22.83 29.83 11.76 31.59 34.07 32.14 29.16 23.05 14.51 20.66 27.77 32.76 32.35 33.42 18.89 16.81 26.40 27.23 17.97 21.12 31.88 27.29 32.64 ----- 32.51 29.11 21.74 31.23
 34.  23.55 40.08 44.77 45.57 44.91 43.28 32.20 47.45 33.76 29.59 43.12 13.06 42.77 45.87 43.18 38.70 30.35 18.16 24.70 35.65 45.82 43.70 46.41 24.42 21.91 35.28 36.03 23.09 28.51 45.74 37.11 46.49 32.51 ----- 39.33 26.56 43.18
 35.  24.76 36.05 40.69 42.30 44.06 37.22 30.70 42.67 30.23 27.94 41.99 12.41 43.71 40.19 39.20 35.76 26.88 17.61 22.06 34.18 43.25 40.20 43.35 24.45 19.82 32.50 33.78 22.87 28.53 42.34 36.84 43.02 29.11 39.33 ----- 25.53 43.11
 36.  15.99 26.95 27.74 27.54 28.73 26.73 21.57 29.40 22.98 21.50 24.47 10.21 26.22 29.01 26.94 26.68 20.61 13.63 17.32 24.12 28.85 29.61 29.97 17.70 15.22 24.63 24.82 16.50 20.16 29.11 23.80 25.94 21.74 26.56 25.53 ----- 28.71
 37.  26.51 38.15 45.14 45.71 49.48 41.48 32.48 48.73 32.61 30.27 45.01 12.45 46.13 45.63 41.99 40.31 28.21 18.09 23.32 37.27 48.92 45.05 47.82 26.66 22.41 36.13 36.06 24.17 30.57 46.21 39.58 46.37 31.23 43.18 43.11 28.71 -----



I don't quite understand the numbers inside the matrix, nor the clustering. What are these very large 2 distant clusters? The positions are those 8,300 of Don or they are other positions? Lc0 should be pretty unrelated to anything conventional A/B engine.
Ferdy
Posts: 4840
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Can the sardine! NNUE clobbers SF.

Post by Ferdy »

Laskos wrote: Sat Jul 18, 2020 11:37 am
Ferdy wrote: Sat Jul 18, 2020 10:27 am
Rebel wrote: Fri Jul 17, 2020 10:24 amThe similarity with the Lc0 related nets would be interesting.
Run sfnn, sf and lc0 blas and added from my old list. Stockfish 2020-07-11 is a dev.

Image

Matrix, engines are run with different movetime. sf10 and sf10 r1 is the same as ref.

I don't quite understand the numbers inside the matrix, nor the clustering. What are these very large 2 distant clusters? The positions are those 8,300 of Don or they are other positions? Lc0 should be pretty unrelated to anything conventional A/B engine.
Plot corrected, I was using an old code :)

The plot is built using scipy at https://docs.scipy.org/doc/scipy/refere ... nkage.html
The two distant clusters was created when using Ward method.

Given simdata from simex2 (Ed), simex2 also creates dendrogram.csv, that csv is then used to plot the dendrogram using https://github.com/fsmosca/Similarity-Dendrogram

Updated plot.
Image
MMarco
Posts: 197
Joined: Sun Apr 12, 2020 1:09 am
Full name: Marc-O Moisan-Plante

Re: Can the sardine! NNUE clobbers SF.

Post by MMarco »

Very interesting.

I noticed that on the second plot SF 10 seems unrelated to any other engine. Is there a particular reason for this?

Thanks!
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: Can the sardine! NNUE clobbers SF.

Post by Laskos »

MMarco wrote: Sat Jul 18, 2020 3:08 pm Very interesting.

I noticed that on the second plot SF 10 seems unrelated to any other engine. Is there a particular reason for this?

Thanks!
Look at his matrix, some weird numbers for Sim. I guess the dendrogram is nonsense too.

I managed to test under SIM the SF NNUE_GK_06-27, it was a location issue, one has to have the executable in the same folder as the Sim tool.

Code: Select all

sim version 3

  Key:

  1) Shredder (time: 100 ms  scale: 1)
  2) Lc0 SV_3010 (time: 100 ms  scale: 1.0)
  3) Lc0 T70 (time: 100 ms  scale: 1.0)
  4) SF_NNUE_bis_location (time: 100 ms  scale: 1)
  5) Stockfish 10 (time: 100 ms  scale: 1)
  6) Stockfish 11 (time: 100 ms  scale: 1)
  7) Stockfish 6 (time: 100 ms  scale: 1)
  8) Stockfish 8 (time: 100 ms  scale: 1)
  9) Stockfish dev (time: 100 ms  scale: 1)
 10) Stockfish dev Bis (time: 100 ms  scale: 1)
 11) 1Andscacs (time: 100 ms  scale: 1)
 12) Komodo 14 (time: 100 ms  scale: 1)
 13) Ethereal (time: 100 ms  scale: 1)

         1     2     3     4     5     6     7     8     9    10    11    12    13
  1.  ----- 36.00 38.76 38.53 40.05 39.80 42.40 40.52 39.04 39.41 40.76 40.64 40.67
  2.  36.00 ----- 67.27 55.32 52.15 52.82 48.94 50.72 53.88 53.34 49.54 49.90 49.07
  3.  38.76 67.27 ----- 55.83 52.62 52.61 49.78 51.07 52.62 52.18 49.73 50.44 50.30
  4.  38.53 55.32 55.83 ----- 55.12 55.38 51.52 54.33 55.80 55.24 52.05 52.11 51.76
  5.  40.05 52.15 52.62 55.12 ----- 64.79 56.43 61.67 63.57 63.51 56.05 55.83 56.25
  6.  39.80 52.82 52.61 55.38 64.79 ----- 55.52 60.31 66.04 65.53 55.55 55.90 54.95
  7.  42.40 48.94 49.78 51.52 56.43 55.52 ----- 60.96 54.79 54.86 56.02 54.90 54.05
  8.  40.52 50.72 51.07 54.33 61.67 60.31 60.96 ----- 59.03 58.62 58.55 56.19 55.45
  9.  39.04 53.88 52.62 55.80 63.57 66.04 54.79 59.03 ----- 66.06 54.88 54.60 53.96
 10.  39.41 53.34 52.18 55.24 63.51 65.53 54.86 58.62 66.06 ----- 54.22 54.90 54.47
 11.  40.76 49.54 49.73 52.05 56.05 55.55 56.02 58.55 54.88 54.22 ----- 53.91 53.44
 12.  40.64 49.90 50.44 52.11 55.83 55.90 54.90 56.19 54.60 54.90 53.91 ----- 53.86
 13.  40.67 49.07 50.30 51.76 56.25 54.95 54.05 55.45 53.96 54.47 53.44 53.86 -----


From the matrix, it is apparent that SF NNUE GK is approximately as close to SF_dev as to Lc0, similarly distanced with respect to them, and is not very similar to anything.


Both correlation and distance methods for cluster give the same clustering shown here:

Image

Which seems to show that SF NNUE is a bit closer to Lc0 than to SF_dev, but not very very close to anything.
smatovic
Posts: 2794
Joined: Wed Mar 10, 2010 10:18 pm
Location: Hamburg, Germany
Full name: Srdja Matovic

Re: Can the sardine! NNUE clobbers SF.

Post by smatovic »

Maybe a depth 1 match between LC0 and NNUE will be useful, to get an idea of how the networks perform against each other, and of what importance the whole search is, or alike.

--
Srdja
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: Can the sardine! NNUE clobbers SF.

Post by Laskos »

smatovic wrote: Sun Jul 19, 2020 9:08 am Maybe a depth 1 match between LC0 and NNUE will be useful, to get an idea of how the networks perform against each other, and of what importance the whole search is, or alike.

--
Srdja
No, the search is still SF. To depth=1 I compared SF NNUE to SF_dev, and SF NNUE is significantly stronger to depth=1:

depth=1
Score of SF_NNUE vs SF_dev: 655 - 265 - 80 [0.695] 1000
Elo difference: 143.1 +/- 22.3, LOS: 100.0 %, DrawRatio: 8.0 %
Finished match

So, the net eval helps a lot at depth=1.
Ferdy
Posts: 4840
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Can the sardine! NNUE clobbers SF.

Post by Ferdy »

MMarco wrote: Sat Jul 18, 2020 3:08 pm Very interesting.

I noticed that on the second plot SF 10 seems unrelated to any other engine. Is there a particular reason for this?

Thanks!
There are some methods for calculating the distance between the newly formed cluster according to https://docs.scipy.org/doc/scipy/refere ... nkage.html.

That one was ward.

This is method single.
PS D:\Chess\Tools\simex2> ./dendrogram.exe --input dendrogram.csv --method single --output single.png
Image

This is method complete.
Image
smatovic
Posts: 2794
Joined: Wed Mar 10, 2010 10:18 pm
Location: Hamburg, Germany
Full name: Srdja Matovic

Re: Can the sardine! NNUE clobbers SF.

Post by smatovic »

Laskos wrote: Sun Jul 19, 2020 9:28 am
smatovic wrote: Sun Jul 19, 2020 9:08 am Maybe a depth 1 match between LC0 and NNUE will be useful, to get an idea of how the networks perform against each other, and of what importance the whole search is, or alike.

--
Srdja
No, the search is still SF. To depth=1 I compared SF NNUE to SF_dev, and SF NNUE is significantly stronger to depth=1:

depth=1
Score of SF_NNUE vs SF_dev: 655 - 265 - 80 [0.695] 1000
Elo difference: 143.1 +/- 22.3, LOS: 100.0 %, DrawRatio: 8.0 %
Finished match

So, the net eval helps a lot at depth=1.
Okay, then we know that NNUE eval is better than SF one, but we do not know what the impact of C/NN size is, and how MCTS-PUCT compares to AB.

--
Srdja