StockFiNN Release and Scaling

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
jjoshua2
Posts: 94
Joined: Sat Mar 10, 2018 5:16 am

StockFiNN Release and Scaling

Post by jjoshua2 » Sat Jul 04, 2020 10:41 pm

This evaluation is created from scratch solely a 3 layer neural network that I started from a random initialization, so while it has search in common with Stockfish the evaluation is not a cloned and tweaked as well like a lot of SF clones that still have NN in the name, and merely mean it has some sort of learning function where it saves positions where it fails low/high. Like Leela, this does all its learning offline, not while playing matches.

Forked on my github from nodchips SF-NNUE repository, working towards symbiotic training with Leelenstein. You can download StockFiNN 0.1 and follow progress here: https://www.patreon.com/posts/stockfinn-0-1-38717611

It is progressing very fast so testing is in flux, but here are some scaling results worth seeing.
Using the default avx2 compile I posted I get about 1.6-1.7M nps with the default 1 thread bench, it's about 40-50% as fast as sf-dev. (There is also an intel skylake bmi2 compile I made) FYI, under a full AVX2 load CPUs get very hot so need good cooling to not downclock.

All testing done with my same 320 game no draw book on 3950x with 1 thread and full 6 man EGTB on nvme ssd. I used Brainfish compile with large pages because it is the fastest Stockfish on my system.

Code: Select all

15s+0.15s
Score of StockFiNN vs BrainFish_200420: 66 - 155 - 99 [0.361]
Elo difference: -99.2 +/- 32.4, LOS: 0.0 %, DrawRatio: 30.9 %

30s+0.3s
Score of StockFiNN vs BrainFish_200420: 72 - 127 - 121 [0.414]
Elo difference: -60.3 +/- 30.2, LOS: 0.0 %, DrawRatio: 37.8 %

60s+0.6s 
Score of StockFiNN vs BrainFish_200420: 74 - 103 - 143 [0.455]
Elo difference: -31.6 +/- 28.4, LOS: 1.5 %, DrawRatio: 44.7 %

120+1.2s 
Score of StockFiNN vs BrainFish_200420: 79 - 75 - 166 [0.506]
Elo difference: 4.3 +/- 26.4, LOS: 62.6 %, DrawRatio: 51.9 %

Code: Select all

fixed 100k nodes 
Score of StockFiNN vs BrainFish_200420: 144 - 95 - 81 [0.577]
Elo difference: 53.6 +/- 33.2, LOS: 99.9 %, DrawRatio: 25.3 %

fixed 1 million nodes 
Score of StockFiNN vs BrainFish_200420: 125 - 66 - 129 [0.592]
Elo difference: 64.8 +/- 29.6, LOS: 100.0 %, DrawRatio: 40.3 %
Testing at 10s+0.1s does not work well as there is too much variance and overhead with initializing NN search. But it seems to scale about 30 elo per doubling better than Stockfish!

Milos
Posts: 3787
Joined: Wed Nov 25, 2009 12:47 am

Re: StockFiNN Release and Scaling

Post by Milos » Sat Jul 04, 2020 10:48 pm

jjoshua2 wrote:
Sat Jul 04, 2020 10:41 pm
This evaluation is created from scratch solely a 3 layer neural network that I started from a random initialization, so while it has search in common with Stockfish the evaluation is not a cloned and tweaked as well like a lot of SF clones that still have NN in the name, and merely mean it has some sort of learning function where it saves positions where it fails low/high. Like Leela, this does all its learning offline, not while playing matches.

Forked on my github from nodchips SF-NNUE repository, working towards symbiotic training with Leelenstein. You can download StockFiNN 0.1 and follow progress here: https://www.patreon.com/posts/stockfinn-0-1-38717611

It is progressing very fast so testing is in flux, but here are some scaling results worth seeing.
Using the default avx2 compile I posted I get about 1.6-1.7M nps with the default 1 thread bench, it's about 40-50% as fast as sf-dev. (There is also an intel skylake bmi2 compile I made) FYI, under a full AVX2 load CPUs get very hot so need good cooling to not downclock.

All testing done with my same 320 game no draw book on 3950x with 1 thread and full 6 man EGTB on nvme ssd. I used Brainfish compile with large pages because it is the fastest Stockfish on my system.

Code: Select all

15s+0.15s
Score of StockFiNN vs BrainFish_200420: 66 - 155 - 99 [0.361]
Elo difference: -99.2 +/- 32.4, LOS: 0.0 %, DrawRatio: 30.9 %

30s+0.3s
Score of StockFiNN vs BrainFish_200420: 72 - 127 - 121 [0.414]
Elo difference: -60.3 +/- 30.2, LOS: 0.0 %, DrawRatio: 37.8 %

60s+0.6s 
Score of StockFiNN vs BrainFish_200420: 74 - 103 - 143 [0.455]
Elo difference: -31.6 +/- 28.4, LOS: 1.5 %, DrawRatio: 44.7 %

120+1.2s 
Score of StockFiNN vs BrainFish_200420: 79 - 75 - 166 [0.506]
Elo difference: 4.3 +/- 26.4, LOS: 62.6 %, DrawRatio: 51.9 %

Code: Select all

fixed 100k nodes 
Score of StockFiNN vs BrainFish_200420: 144 - 95 - 81 [0.577]
Elo difference: 53.6 +/- 33.2, LOS: 99.9 %, DrawRatio: 25.3 %

fixed 1 million nodes 
Score of StockFiNN vs BrainFish_200420: 125 - 66 - 129 [0.592]
Elo difference: 64.8 +/- 29.6, LOS: 100.0 %, DrawRatio: 40.3 %
Testing at 10s+0.1s does not work well as there is too much variance and overhead with initializing NN search. But it seems to scale about 30 elo per doubling better than Stockfish!
What's the point of yet another fork of SF-NNUE? Any actual changes in NN or search to accommodate it, or TM?

Raphexon
Posts: 274
Joined: Sun Mar 17, 2019 11:00 am
Full name: Henk Drost

Re: StockFiNN Release and Scaling

Post by Raphexon » Sat Jul 04, 2020 11:02 pm

Milos wrote:
Sat Jul 04, 2020 10:48 pm
jjoshua2 wrote:
Sat Jul 04, 2020 10:41 pm
This evaluation is created from scratch solely a 3 layer neural network that I started from a random initialization, so while it has search in common with Stockfish the evaluation is not a cloned and tweaked as well like a lot of SF clones that still have NN in the name, and merely mean it has some sort of learning function where it saves positions where it fails low/high. Like Leela, this does all its learning offline, not while playing matches.

Forked on my github from nodchips SF-NNUE repository, working towards symbiotic training with Leelenstein. You can download StockFiNN 0.1 and follow progress here: https://www.patreon.com/posts/stockfinn-0-1-38717611

It is progressing very fast so testing is in flux, but here are some scaling results worth seeing.
Using the default avx2 compile I posted I get about 1.6-1.7M nps with the default 1 thread bench, it's about 40-50% as fast as sf-dev. (There is also an intel skylake bmi2 compile I made) FYI, under a full AVX2 load CPUs get very hot so need good cooling to not downclock.

All testing done with my same 320 game no draw book on 3950x with 1 thread and full 6 man EGTB on nvme ssd. I used Brainfish compile with large pages because it is the fastest Stockfish on my system.

Code: Select all

15s+0.15s
Score of StockFiNN vs BrainFish_200420: 66 - 155 - 99 [0.361]
Elo difference: -99.2 +/- 32.4, LOS: 0.0 %, DrawRatio: 30.9 %

30s+0.3s
Score of StockFiNN vs BrainFish_200420: 72 - 127 - 121 [0.414]
Elo difference: -60.3 +/- 30.2, LOS: 0.0 %, DrawRatio: 37.8 %

60s+0.6s 
Score of StockFiNN vs BrainFish_200420: 74 - 103 - 143 [0.455]
Elo difference: -31.6 +/- 28.4, LOS: 1.5 %, DrawRatio: 44.7 %

120+1.2s 
Score of StockFiNN vs BrainFish_200420: 79 - 75 - 166 [0.506]
Elo difference: 4.3 +/- 26.4, LOS: 62.6 %, DrawRatio: 51.9 %

Code: Select all

fixed 100k nodes 
Score of StockFiNN vs BrainFish_200420: 144 - 95 - 81 [0.577]
Elo difference: 53.6 +/- 33.2, LOS: 99.9 %, DrawRatio: 25.3 %

fixed 1 million nodes 
Score of StockFiNN vs BrainFish_200420: 125 - 66 - 129 [0.592]
Elo difference: 64.8 +/- 29.6, LOS: 100.0 %, DrawRatio: 40.3 %
Testing at 10s+0.1s does not work well as there is too much variance and overhead with initializing NN search. But it seems to scale about 30 elo per doubling better than Stockfish!
What's the point of yet another fork of SF-NNUE? Any actual changes in NN or search to accommodate it, or TM?
Only difference is netsize, which is literally changing a single number in the source.

lkaufman
Posts: 4290
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: StockFiNN Release and Scaling

Post by lkaufman » Sat Jul 04, 2020 11:06 pm

jjoshua2 wrote:
Sat Jul 04, 2020 10:41 pm
This evaluation is created from scratch solely a 3 layer neural network that I started from a random initialization, so while it has search in common with Stockfish the evaluation is not a cloned and tweaked as well like a lot of SF clones that still have NN in the name, and merely mean it has some sort of learning function where it saves positions where it fails low/high. Like Leela, this does all its learning offline, not while playing matches.

Forked on my github from nodchips SF-NNUE repository, working towards symbiotic training with Leelenstein. You can download StockFiNN 0.1 and follow progress here: https://www.patreon.com/posts/stockfinn-0-1-38717611

It is progressing very fast so testing is in flux, but here are some scaling results worth seeing.
Using the default avx2 compile I posted I get about 1.6-1.7M nps with the default 1 thread bench, it's about 40-50% as fast as sf-dev. (There is also an intel skylake bmi2 compile I made) FYI, under a full AVX2 load CPUs get very hot so need good cooling to not downclock.

All testing done with my same 320 game no draw book on 3950x with 1 thread and full 6 man EGTB on nvme ssd. I used Brainfish compile with large pages because it is the fastest Stockfish on my system.

Code: Select all

15s+0.15s
Score of StockFiNN vs BrainFish_200420: 66 - 155 - 99 [0.361]
Elo difference: -99.2 +/- 32.4, LOS: 0.0 %, DrawRatio: 30.9 %

30s+0.3s
Score of StockFiNN vs BrainFish_200420: 72 - 127 - 121 [0.414]
Elo difference: -60.3 +/- 30.2, LOS: 0.0 %, DrawRatio: 37.8 %

60s+0.6s 
Score of StockFiNN vs BrainFish_200420: 74 - 103 - 143 [0.455]
Elo difference: -31.6 +/- 28.4, LOS: 1.5 %, DrawRatio: 44.7 %

120+1.2s 
Score of StockFiNN vs BrainFish_200420: 79 - 75 - 166 [0.506]
Elo difference: 4.3 +/- 26.4, LOS: 62.6 %, DrawRatio: 51.9 %

Code: Select all

fixed 100k nodes 
Score of StockFiNN vs BrainFish_200420: 144 - 95 - 81 [0.577]
Elo difference: 53.6 +/- 33.2, LOS: 99.9 %, DrawRatio: 25.3 %

fixed 1 million nodes 
Score of StockFiNN vs BrainFish_200420: 125 - 66 - 129 [0.592]
Elo difference: 64.8 +/- 29.6, LOS: 100.0 %, DrawRatio: 40.3 %
Testing at 10s+0.1s does not work well as there is too much variance and overhead with initializing NN search. But it seems to scale about 30 elo per doubling better than Stockfish!
I know of opening books with low draw rates, but what is a "no draw book"?
Komodo rules!

jjoshua2
Posts: 94
Joined: Sat Mar 10, 2018 5:16 am

Re: StockFiNN Release and Scaling

Post by jjoshua2 » Sat Jul 04, 2020 11:15 pm

It's a low draw book I made for Leelenstein playing two variants with a full opening book and then taking out all the games that ended in draws. It ends up getting a nice split between draw, won, and decisive games that has worked for me to maximize elo spreads while still being representative. It might bias results for testing Leelenstein vs SF, but works great for testing engines against themselves or similar.

jjoshua2
Posts: 94
Joined: Sat Mar 10, 2018 5:16 am

Re: StockFiNN Release and Scaling

Post by jjoshua2 » Sat Jul 04, 2020 11:18 pm

Milos wrote:
Sat Jul 04, 2020 10:48 pm
What's the point of yet another fork of SF-NNUE? Any actual changes in NN or search to accommodate it, or TM?
The engine fork is not the interesting part. The interesting part is that I have trained a net that is as strong as Stockfish itself in a short TC 1 thread, and very rapidly without too many resources. And still gaining a lot

Milos
Posts: 3787
Joined: Wed Nov 25, 2009 12:47 am

Re: StockFiNN Release and Scaling

Post by Milos » Sat Jul 04, 2020 11:47 pm

jjoshua2 wrote:
Sat Jul 04, 2020 11:18 pm
Milos wrote:
Sat Jul 04, 2020 10:48 pm
What's the point of yet another fork of SF-NNUE? Any actual changes in NN or search to accommodate it, or TM?
The engine fork is not the interesting part. The interesting part is that I have trained a net that is as strong as Stockfish itself in a short TC 1 thread, and very rapidly without too many resources. And still gaining a lot
I see, but net itself is only different to SF-NNUE one in size right?
So can actual binary accept different net sizes (like it's the case with Lc0)? Would be good to have a unique version of main binary that is up to date with SFdev and that accepts different nets.

Chessqueen
Posts: 1043
Joined: Wed Sep 05, 2018 12:16 am
Full name: Nancy M Pichardo

Re: StockFiNN Release and Scaling

Post by Chessqueen » Sun Jul 05, 2020 12:00 am

jjoshua2 wrote:
Sat Jul 04, 2020 11:18 pm
Milos wrote:
Sat Jul 04, 2020 10:48 pm
What's the point of yet another fork of SF-NNUE? Any actual changes in NN or search to accommodate it, or TM?
The engine fork is not the interesting part. The interesting part is that I have trained a net that is as strong as Stockfish itself in a short TC 1 thread, and very rapidly without too many resources. And still gaining a lot
So you are saying that this StockFINN might be the Next TCEC Champion even better than the latest Stockfish that beat LCO a few days ago :?:

Raphexon
Posts: 274
Joined: Sun Mar 17, 2019 11:00 am
Full name: Henk Drost

Re: StockFiNN Release and Scaling

Post by Raphexon » Sun Jul 05, 2020 1:23 am

jjoshua2 wrote:
Sat Jul 04, 2020 11:18 pm
Milos wrote:
Sat Jul 04, 2020 10:48 pm
What's the point of yet another fork of SF-NNUE? Any actual changes in NN or search to accommodate it, or TM?
The engine fork is not the interesting part. The interesting part is that I have trained a net that is as strong as Stockfish itself in a short TC 1 thread, and very rapidly without too many resources. And still gaining a lot
How much data did you even need in the end?
At least I'm glad that somebody you have surpassed my 26-6 and 27-6 nets now in little time.

Hope I can get my stumbles with the latest binaries quickly fixed so I can upgrade mine too.

jjoshua2
Posts: 94
Joined: Sat Mar 10, 2018 5:16 am

Re: StockFiNN Release and Scaling

Post by jjoshua2 » Sun Jul 05, 2020 2:11 am

Important point I forget to state before was that all my tests were using slowmover 60 and defaults of course for Stockfish. My last net had about 950 million positions and trained for maybe 5 billion steps/positions (can't remember)? I'm training my next net a lot slower, almost 7 billion to first learning rate drop.

Post Reply