Best Nets Made from Human Games Data

[BrendanJNorman]

I guess that there are very few (if any) nets made exclusively from human games. I guess you could build the nets based on the win/loss/draw alone.
You would need a lot of statistics to create such a net.

The Lichess games would be a good candidate for that, because the volume is so high.
I use a formula like this to estimate centipawns, based on wins/losses/draws:
UPDATE Epd SET coef = (white_wins - black_wins)* 1.0/(1e-20+white_wins+black_wins+draws)
UPDATE Epd SET coef = -coef WHERE Epd like '% b %'
and then:
cce = round(coef * 444.0,0)
It's only a good fit for a score of +/- one full piece. At 444 or -444 it is all wins or losses.

I would not want to use the data if there are less than 32 decisive games for the position.
Also, I would use only games where both players were at or above 2500.
Because there are so many Lichess games, that will still leave you more than ten million games from which to draw analysis.

From Kingbase with material qsearch mixed in. Uses the original SF nnue format. Could use some better, deeper material search.

https://www.patreon.com/posts/harmon-nnue-44549655

Yes, this one plays great chess!

[BrendanJNorman]

There is Maia Chess:

https://www.chessprogramming.org/Maia_Chess
https://maiachess.com/

--
Srdja

Too weak. Tops at only 1900.

I wanna see a net that's 2200-2500 and "feels" as human (or more) than Maia.

[BrendanJNorman]

There are two issues with human games for training: first, humans blunder a lot. Even strong players. Secondly, you won't get enough games that way. You need billions of positions to get a well-trained net.

Could be possible. A bit of crowd sourced help would be useful (supply of cores).
I would guess there are enough human-human games on LiChess of "reasonable" strength players.
Positions could be blunderchecked by SF or whatever, that's the time consuming part.
A 50 core machine can blunder check about 5000 games per second at d9 (which would be enough, probably) = 400M positions a day. 10B positions would take 25 days per 50 core machine. Any core volunteers? I'll organise it.

THIS would be interesting.

[carldaman]

Texium has posted this link to kennyfrc's nets, built from human games, elsewhere:

https://github.com/kennyfrc/nnue-networks/releases

What's not clear to me is what engine can be used to run such nets - any ideas?

[sarona]

He was using his own CFish branches with these nets. Here are three. You have to compile the code.

https://github.com/kennyfrc/Cfish/tree/elo2300
https://github.com/kennyfrc/Cfish/tree/elo1600
https://github.com/kennyfrc/Cfish/tree/elo1200

[carldaman]

Texium also mentioned that the kennyfrc nets could be used with Rodent-NNUE, by renaming the original net, and also with any Stockfish build that can handle 20 GB nets.

[BrendanJNorman]

He was using his own CFish branches with these nets. Here are three. You have to compile the code.

https://github.com/kennyfrc/Cfish/tree/elo2300
https://github.com/kennyfrc/Cfish/tree/elo1600
https://github.com/kennyfrc/Cfish/tree/elo1200

Is anyone willing to compile these? I have no idea how.

[adnoh]

He was using his own CFish branches with these nets. Here are three. You have to compile the code.

https://github.com/kennyfrc/Cfish/tree/elo2300
https://github.com/kennyfrc/Cfish/tree/elo1600
https://github.com/kennyfrc/Cfish/tree/elo1200

Is anyone willing to compile these? I have no idea how.

I tried to compile them but ran into 2 problems.
1) Using current compiler it failed with an error. It is possible I am missing some packages but I do not know.
2) Perhaps more importantly, the NN net the makefile refers to are no longer available anyways.

Code: Select all

[D:\Users\Cfish]make build
Default net: kb2300-20201231-1147.bin
Downloading https://github.com/kennyfrc/nnue-networks/releases/download/early-release/kb2300-20201231-1147.bin

Config:
debug: 'no'
sanitize: 'no'
optimize: 'yes'
arch: 'x86_64'
bits: '64'
kernel: 'MSYS_NT-10.0-22621'
os: 'Windows_NT'
prefetch: 'yes'
popcnt: 'yes'
pext: 'yes'
sse: 'yes'
mmx: 'no'
sse2: 'yes'
ssse3: 'yes'
sse41: 'yes'
avx: 'yes'
avx2: 'yes'
avx512: 'no'
vnni: 'no'
neon: 'no'
native: 'yes'
embed: 'no'

Flags:
CC: gcc
CFLAGS: -Wall -std=c11  -pedantic -Wextra -Wshadow -m64 -Wno-pedantic-ms-format -DNDEBUG -O3 -DIS_64BIT -msse3 -mpopcnt -DUSE_POPCNT -DUSE_PEXT -msse4 -mbmi2 -DUSE_AVX2 -mavx2 -DUSE_SSE41 -msse4.1
 -DUSE_SSSE3 -mssse3 -DUSE_SSE2 -msse2 -msse -DUSE_SSE -march=native -DNUMA -DNNUE
LDFLAGS:  -lm -m64 -Wl,--no-as-needed -lpthread

Testing config sanity. If this fails, try 'make help' ...

make ARCH=auto COMP=gcc all
make[1]: Entering directory '/d/Users/Cfish'
gcc -Wall -std=c11  -pedantic -Wextra -Wshadow -m64 -Wno-pedantic-ms-format -DNDEBUG -O3 -DIS_64BIT -msse3 -mpopcnt -DUSE_POPCNT -DUSE_PEXT -msse4 -mbmi2 -DUSE_AVX2 -mavx2 -DUSE_SSE41 -msse4.1 -DU
SE_SSSE3 -mssse3 -DUSE_SSE2 -msse2 -msse -DUSE_SSE -march=native -DNUMA -DNNUE   -c -o benchmark.o benchmark.c
In file included from C:/msys64/ucrt64/include/minwindef.h:163,
                 from C:/msys64/ucrt64/include/windef.h:9,
                 from C:/msys64/ucrt64/include/windows.h:69,
                 from types.h:53,
                 from evaluate.h:4,
                 from benchmark.c:26:
thread.h:50:62: error: expected identifier before '(' token
   50 |   THREAD_SLEEP, THREAD_SEARCH, THREAD_TT_CLEAR, THREAD_EXIT, THREAD_RESUME
      |                                                              ^~~~~~~~~~~~~
make[1]: *** [<builtin>: benchmark.o] Error 1
make[1]: Leaving directory '/d/Users/Cfish'
make: *** [Makefile:848: build] Error 2

[D:\Users\Cfish]

So noting that the developer of those nets himself deprecated them and suggested a newer net, I downloaded the latest cfish (cfish 12) and ran it with the recommended nn-maia-distilled-0209-final.bin net and it works fine.

https://github.com/syzygy1/Cfish

https://github.com/kennyfrc/nnue-networks/releases

Note the developer states the following:

Distilled Maia Net Latest
Download this instead of the previous nets as the past ones have issues with low evals. This one has more reasonable evaluations. Trained with 30m positions of the maia1900 leela network (self-play at 1 node).

About equal to ~2200 elo. When nodestime is set to 1, it's about ~1200 elo.

Best used with Cfish set to pure=yes.

OK. So I ran a quick 500 game match with this combination of cfish and net against OpenTal 1.0 and got the following results so it is pretty strong without being >3000 level strong. Note make sure you set the UCI option of Use NNUE to "Pure" as the developer recommends.

Code: Select all

Score of Cfish 12 - nn-maia-distilled-0209-final.bin vs OpenTal 1.0: 317 - 164 - 19 [0.653]
...      Cfish 12 - nn-maia-distilled-0209-final.bin playing White: 172 - 67 - 11  [0.710] 250
...      Cfish 12 - nn-maia-distilled-0209-final.bin playing Black: 145 - 97 - 8  [0.596] 250
...      White vs Black: 269 - 212 - 19  [0.557] 500
Elo difference: 109.8 +/- 31.4, LOS: 100.0 %, DrawRatio: 3.8 %
500 of 500 games finished.

Player: Cfish 12 - nn-maia-distilled-0209-final.bin
   "Draw by 3-fold repetition": 5
   "Draw by adjudication": 4
   "Draw by adjudication: SyzygyTB": 10
   "Loss: Black wins by adjudication": 67
   "Loss: White mates": 1
   "Loss: White wins by adjudication": 96
   "Win: Black mates": 1
   "Win: Black wins by adjudication": 67
   "Win: Black wins by adjudication: SyzygyTB": 77
   "Win: White mates": 4
   "Win: White wins by adjudication": 95
   "Win: White wins by adjudication: SyzygyTB": 73
Player: OpenTal 1.0
   "Draw by 3-fold repetition": 5
   "Draw by adjudication": 4
   "Draw by adjudication: SyzygyTB": 10
   "Loss: Black mates": 1
   "Loss: Black wins by adjudication": 67
   "Loss: Black wins by adjudication: SyzygyTB": 77
   "Loss: White mates": 4
   "Loss: White wins by adjudication": 95
   "Loss: White wins by adjudication: SyzygyTB": 73
   "Win: Black wins by adjudication": 67
   "Win: White mates": 1
   "Win: White wins by adjudication": 96

I also ran Stefan Pohl's EAS tool on the PGN and although not "reliable", it is more aggressive than OpenTal 1.0?

Code: Select all

***************************************************************************** 
*** Engine Aggressiveness Tool V5.21 Score points Ratinglist 
***************************************************************************** 
*** Meanwhile, the scoring-system of the EAS-Tool got really complex, so 
*** please check out the ReadMe-file, where you find the explanation... 
***************************************************************************** 
*** Evaluated file: a.pgn 
***************************************************************************** 
                                 bad  avg.win 
Rank  EAS-Score  sacs   shorts  draws  moves  Engine/player 
-------------------------------------------------------------------
   1     89099  15.14%  04.10%  15.79%   51   Cfish 12 - nn-maia-distilled-0209-final.bin   XXXXX WARNING: Not enough games, EAS-score not reliable [50+ wins and 30+ draws needed] XXXXX  
   2     73583  13.41%  19.51%  31.58%   40   OpenTal 1.0   XXXXX WARNING: Not enough games, EAS-score not reliable [50+ wins and 30+ draws needed] XXXXX  
-------------------------------------------------------------------
*** Average length of all won games:     47 moves
*** Engine gets bonuspoints, if its avg. won games length is shorter
*** Engine gets maluspoints, if its avg. won games length is longer
***************************************************************************** 
***************************************************************************** 
***************************************************************************** 
*** 2nd Ratinglist with more stats in percent-values ************************ 
***************************************************************************** 
*** Average length of all won games:  47 moves 
*** Calculated limit for short wins giving EAS-points: 30 moves 
***************************************************************************** 
                       avg.win                                                                                                                          bad   
Rank  EAS-Score   wins  moves   sacs    sacsQ    sacs5+   sacs4    sacs3    sacs2    sacs1    all shorts short10  short15  short20  short25  short30   draws    Engine/player
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   1     89099      317   51   15.14% =[00.00% + 00.32% + 00.95% + 01.58% + 05.36% + 06.94%]    04.10% = [00.00% + 00.00% + 00.00% + 01.26% + 02.84%]  15.79%   Cfish 12 - nn-maia-distilled-0209-final.bin  
   2     73583      164   40   13.41% =[00.00% + 00.00% + 00.00% + 01.83% + 04.27% + 07.32%]    19.51% = [00.00% + 00.00% + 00.61% + 06.10% + 12.80%]  31.58%   OpenTal 1.0  
**************************************************** 
*** EAS-tool (C) 2022 Stefan Pohl (www.sp-cc.de) *** 
**************************************************** 

The PGN from the 500 game match and the EAS extracted PGNs are all here if you like to have a look and see whether the style and strength suites what you are looking for.

https://1drv.ms/f/s!Aqe2-2i2H5hegoJ7XbZ ... Q?e=1ScNp8

[BluRayVertex]

i also happen to now have some leela nets trained on 2500 elo rated lichess players, in a maia style net, and a 128x10 net trained on the entire lichess elite database

[BluRayVertex]

despite being trained on lichess games it is probably the best net for predicting the moves of strong gms OTB, given you're looking in nibbler at top moves and policy moves