Best Nets Made from Human Games Data

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

BrendanJNorman
Posts: 2564
Joined: Mon Feb 08, 2016 12:43 am
Full name: Brendan J Norman

Re: Best Nets Made from Human Games Data

Post by BrendanJNorman »

dkappe wrote: Fri Jun 09, 2023 6:04 am
Dann Corbit wrote: Thu Jun 08, 2023 4:44 am I guess that there are very few (if any) nets made exclusively from human games. I guess you could build the nets based on the win/loss/draw alone.
You would need a lot of statistics to create such a net.

The Lichess games would be a good candidate for that, because the volume is so high.
I use a formula like this to estimate centipawns, based on wins/losses/draws:
UPDATE Epd SET coef = (white_wins - black_wins)* 1.0/(1e-20+white_wins+black_wins+draws)
UPDATE Epd SET coef = -coef WHERE Epd like '% b %'
and then:
cce = round(coef * 444.0,0)
It's only a good fit for a score of +/- one full piece. At 444 or -444 it is all wins or losses.

I would not want to use the data if there are less than 32 decisive games for the position.
Also, I would use only games where both players were at or above 2500.
Because there are so many Lichess games, that will still leave you more than ten million games from which to draw analysis.
From Kingbase with material qsearch mixed in. Uses the original SF nnue format. Could use some better, deeper material search.

https://www.patreon.com/posts/harmon-nnue-44549655
Yes, this one plays great chess!
BrendanJNorman
Posts: 2564
Joined: Mon Feb 08, 2016 12:43 am
Full name: Brendan J Norman

Re: Best Nets Made from Human Games Data

Post by BrendanJNorman »

smatovic wrote: Thu Jun 08, 2023 6:19 am There is Maia Chess:

https://www.chessprogramming.org/Maia_Chess
https://maiachess.com/

--
Srdja
Too weak. Tops at only 1900.

I wanna see a net that's 2200-2500 and "feels" as human (or more) than Maia.
BrendanJNorman
Posts: 2564
Joined: Mon Feb 08, 2016 12:43 am
Full name: Brendan J Norman

Re: Best Nets Made from Human Games Data

Post by BrendanJNorman »

chrisw wrote: Sat Jun 10, 2023 7:47 pm
jdart wrote: Sat Jun 10, 2023 4:35 pm There are two issues with human games for training: first, humans blunder a lot. Even strong players. Secondly, you won't get enough games that way. You need billions of positions to get a well-trained net.

Could be possible. A bit of crowd sourced help would be useful (supply of cores).
I would guess there are enough human-human games on LiChess of "reasonable" strength players.
Positions could be blunderchecked by SF or whatever, that's the time consuming part.
A 50 core machine can blunder check about 5000 games per second at d9 (which would be enough, probably) = 400M positions a day. 10B positions would take 25 days per 50 core machine. Any core volunteers? I'll organise it.
THIS would be interesting.
carldaman
Posts: 2283
Joined: Sat Jun 02, 2012 2:13 am

Re: Best Nets Made from Human Games Data

Post by carldaman »

Texium has posted this link to kennyfrc's nets, built from human games, elsewhere:

https://github.com/kennyfrc/nnue-networks/releases

What's not clear to me is what engine can be used to run such nets - any ideas?
sarona
Posts: 122
Joined: Tue Oct 29, 2019 4:14 pm
Location: Canada
Full name: Ron Doughie

Re: Best Nets Made from Human Games Data

Post by sarona »

He was using his own CFish branches with these nets. Here are three. You have to compile the code.

https://github.com/kennyfrc/Cfish/tree/elo2300
https://github.com/kennyfrc/Cfish/tree/elo1600
https://github.com/kennyfrc/Cfish/tree/elo1200
carldaman
Posts: 2283
Joined: Sat Jun 02, 2012 2:13 am

Re: Best Nets Made from Human Games Data

Post by carldaman »

Texium also mentioned that the kennyfrc nets could be used with Rodent-NNUE, by renaming the original net, and also with any Stockfish build that can handle 20 GB nets.
BrendanJNorman
Posts: 2564
Joined: Mon Feb 08, 2016 12:43 am
Full name: Brendan J Norman

Re: Best Nets Made from Human Games Data

Post by BrendanJNorman »

sarona wrote: Mon Jun 12, 2023 12:30 am He was using his own CFish branches with these nets. Here are three. You have to compile the code.

https://github.com/kennyfrc/Cfish/tree/elo2300
https://github.com/kennyfrc/Cfish/tree/elo1600
https://github.com/kennyfrc/Cfish/tree/elo1200
Is anyone willing to compile these? I have no idea how. :lol: :roll:
adnoh
Posts: 75
Joined: Tue Jun 26, 2007 6:31 am
Full name: Charles Wong

Re: Best Nets Made from Human Games Data

Post by adnoh »

BrendanJNorman wrote: Tue Jun 13, 2023 9:55 am
sarona wrote: Mon Jun 12, 2023 12:30 am He was using his own CFish branches with these nets. Here are three. You have to compile the code.

https://github.com/kennyfrc/Cfish/tree/elo2300
https://github.com/kennyfrc/Cfish/tree/elo1600
https://github.com/kennyfrc/Cfish/tree/elo1200
Is anyone willing to compile these? I have no idea how. :lol: :roll:
I tried to compile them but ran into 2 problems.
1) Using current compiler it failed with an error. It is possible I am missing some packages but I do not know.
2) Perhaps more importantly, the NN net the makefile refers to are no longer available anyways.

Code: Select all

[D:\Users\Cfish]make build
Default net: kb2300-20201231-1147.bin
Downloading https://github.com/kennyfrc/nnue-networks/releases/download/early-release/kb2300-20201231-1147.bin

Config:
debug: 'no'
sanitize: 'no'
optimize: 'yes'
arch: 'x86_64'
bits: '64'
kernel: 'MSYS_NT-10.0-22621'
os: 'Windows_NT'
prefetch: 'yes'
popcnt: 'yes'
pext: 'yes'
sse: 'yes'
mmx: 'no'
sse2: 'yes'
ssse3: 'yes'
sse41: 'yes'
avx: 'yes'
avx2: 'yes'
avx512: 'no'
vnni: 'no'
neon: 'no'
native: 'yes'
embed: 'no'

Flags:
CC: gcc
CFLAGS: -Wall -std=c11  -pedantic -Wextra -Wshadow -m64 -Wno-pedantic-ms-format -DNDEBUG -O3 -DIS_64BIT -msse3 -mpopcnt -DUSE_POPCNT -DUSE_PEXT -msse4 -mbmi2 -DUSE_AVX2 -mavx2 -DUSE_SSE41 -msse4.1
 -DUSE_SSSE3 -mssse3 -DUSE_SSE2 -msse2 -msse -DUSE_SSE -march=native -DNUMA -DNNUE
LDFLAGS:  -lm -m64 -Wl,--no-as-needed -lpthread

Testing config sanity. If this fails, try 'make help' ...

make ARCH=auto COMP=gcc all
make[1]: Entering directory '/d/Users/Cfish'
gcc -Wall -std=c11  -pedantic -Wextra -Wshadow -m64 -Wno-pedantic-ms-format -DNDEBUG -O3 -DIS_64BIT -msse3 -mpopcnt -DUSE_POPCNT -DUSE_PEXT -msse4 -mbmi2 -DUSE_AVX2 -mavx2 -DUSE_SSE41 -msse4.1 -DU
SE_SSSE3 -mssse3 -DUSE_SSE2 -msse2 -msse -DUSE_SSE -march=native -DNUMA -DNNUE   -c -o benchmark.o benchmark.c
In file included from C:/msys64/ucrt64/include/minwindef.h:163,
                 from C:/msys64/ucrt64/include/windef.h:9,
                 from C:/msys64/ucrt64/include/windows.h:69,
                 from types.h:53,
                 from evaluate.h:4,
                 from benchmark.c:26:
thread.h:50:62: error: expected identifier before '(' token
   50 |   THREAD_SLEEP, THREAD_SEARCH, THREAD_TT_CLEAR, THREAD_EXIT, THREAD_RESUME
      |                                                              ^~~~~~~~~~~~~
make[1]: *** [<builtin>: benchmark.o] Error 1
make[1]: Leaving directory '/d/Users/Cfish'
make: *** [Makefile:848: build] Error 2

[D:\Users\Cfish]
So noting that the developer of those nets himself deprecated them and suggested a newer net, I downloaded the latest cfish (cfish 12) and ran it with the recommended nn-maia-distilled-0209-final.bin net and it works fine.

https://github.com/syzygy1/Cfish

https://github.com/kennyfrc/nnue-networks/releases

Note the developer states the following:
Distilled Maia Net Latest
Download this instead of the previous nets as the past ones have issues with low evals. This one has more reasonable evaluations. Trained with 30m positions of the maia1900 leela network (self-play at 1 node).

About equal to ~2200 elo. When nodestime is set to 1, it's about ~1200 elo.

Best used with Cfish set to pure=yes.
OK. So I ran a quick 500 game match with this combination of cfish and net against OpenTal 1.0 and got the following results so it is pretty strong without being >3000 level strong. Note make sure you set the UCI option of Use NNUE to "Pure" as the developer recommends.

Code: Select all

Score of Cfish 12 - nn-maia-distilled-0209-final.bin vs OpenTal 1.0: 317 - 164 - 19 [0.653]
...      Cfish 12 - nn-maia-distilled-0209-final.bin playing White: 172 - 67 - 11  [0.710] 250
...      Cfish 12 - nn-maia-distilled-0209-final.bin playing Black: 145 - 97 - 8  [0.596] 250
...      White vs Black: 269 - 212 - 19  [0.557] 500
Elo difference: 109.8 +/- 31.4, LOS: 100.0 %, DrawRatio: 3.8 %
500 of 500 games finished.

Player: Cfish 12 - nn-maia-distilled-0209-final.bin
   "Draw by 3-fold repetition": 5
   "Draw by adjudication": 4
   "Draw by adjudication: SyzygyTB": 10
   "Loss: Black wins by adjudication": 67
   "Loss: White mates": 1
   "Loss: White wins by adjudication": 96
   "Win: Black mates": 1
   "Win: Black wins by adjudication": 67
   "Win: Black wins by adjudication: SyzygyTB": 77
   "Win: White mates": 4
   "Win: White wins by adjudication": 95
   "Win: White wins by adjudication: SyzygyTB": 73
Player: OpenTal 1.0
   "Draw by 3-fold repetition": 5
   "Draw by adjudication": 4
   "Draw by adjudication: SyzygyTB": 10
   "Loss: Black mates": 1
   "Loss: Black wins by adjudication": 67
   "Loss: Black wins by adjudication: SyzygyTB": 77
   "Loss: White mates": 4
   "Loss: White wins by adjudication": 95
   "Loss: White wins by adjudication: SyzygyTB": 73
   "Win: Black wins by adjudication": 67
   "Win: White mates": 1
   "Win: White wins by adjudication": 96
I also ran Stefan Pohl's EAS tool on the PGN and although not "reliable", it is more aggressive than OpenTal 1.0?

Code: Select all

***************************************************************************** 
*** Engine Aggressiveness Tool V5.21 Score points Ratinglist 
***************************************************************************** 
*** Meanwhile, the scoring-system of the EAS-Tool got really complex, so 
*** please check out the ReadMe-file, where you find the explanation... 
***************************************************************************** 
*** Evaluated file: a.pgn 
***************************************************************************** 
                                 bad  avg.win 
Rank  EAS-Score  sacs   shorts  draws  moves  Engine/player 
-------------------------------------------------------------------
   1     89099  15.14%  04.10%  15.79%   51   Cfish 12 - nn-maia-distilled-0209-final.bin   XXXXX WARNING: Not enough games, EAS-score not reliable [50+ wins and 30+ draws needed] XXXXX  
   2     73583  13.41%  19.51%  31.58%   40   OpenTal 1.0   XXXXX WARNING: Not enough games, EAS-score not reliable [50+ wins and 30+ draws needed] XXXXX  
-------------------------------------------------------------------
*** Average length of all won games:     47 moves
*** Engine gets bonuspoints, if its avg. won games length is shorter
*** Engine gets maluspoints, if its avg. won games length is longer
***************************************************************************** 
***************************************************************************** 
***************************************************************************** 
*** 2nd Ratinglist with more stats in percent-values ************************ 
***************************************************************************** 
*** Average length of all won games:  47 moves 
*** Calculated limit for short wins giving EAS-points: 30 moves 
***************************************************************************** 
                       avg.win                                                                                                                          bad   
Rank  EAS-Score   wins  moves   sacs    sacsQ    sacs5+   sacs4    sacs3    sacs2    sacs1    all shorts short10  short15  short20  short25  short30   draws    Engine/player
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   1     89099      317   51   15.14% =[00.00% + 00.32% + 00.95% + 01.58% + 05.36% + 06.94%]    04.10% = [00.00% + 00.00% + 00.00% + 01.26% + 02.84%]  15.79%   Cfish 12 - nn-maia-distilled-0209-final.bin  
   2     73583      164   40   13.41% =[00.00% + 00.00% + 00.00% + 01.83% + 04.27% + 07.32%]    19.51% = [00.00% + 00.00% + 00.61% + 06.10% + 12.80%]  31.58%   OpenTal 1.0  
**************************************************** 
*** EAS-tool (C) 2022 Stefan Pohl (www.sp-cc.de) *** 
**************************************************** 
The PGN from the 500 game match and the EAS extracted PGNs are all here if you like to have a look and see whether the style and strength suites what you are looking for.

https://1drv.ms/f/s!Aqe2-2i2H5hegoJ7XbZ ... Q?e=1ScNp8
BluRayVertex
Posts: 4
Joined: Fri Oct 20, 2023 4:52 am
Full name: Nick V

Re: Best Nets Made from Human Games Data

Post by BluRayVertex »

i also happen to now have some leela nets trained on 2500 elo rated lichess players, in a maia style net, and a 128x10 net trained on the entire lichess elite database
BluRayVertex
Posts: 4
Joined: Fri Oct 20, 2023 4:52 am
Full name: Nick V

Re: Best Nets Made from Human Games Data

Post by BluRayVertex »

despite being trained on lichess games it is probably the best net for predicting the moves of strong gms OTB, given you're looking in nibbler at top moves and policy moves