StockFiNN Release and Scaling

[jjoshua2]

This evaluation is created from scratch solely a 3 layer neural network that I started from a random initialization, so while it has search in common with Stockfish the evaluation is not a cloned and tweaked as well like a lot of SF clones that still have NN in the name, and merely mean it has some sort of learning function where it saves positions where it fails low/high. Like Leela, this does all its learning offline, not while playing matches.

Forked on my github from nodchips SF-NNUE repository, working towards symbiotic training with Leelenstein. You can download StockFiNN 0.1 and follow progress here: https://www.patreon.com/posts/stockfinn-0-1-38717611

It is progressing very fast so testing is in flux, but here are some scaling results worth seeing.
Using the default avx2 compile I posted I get about 1.6-1.7M nps with the default 1 thread bench, it's about 40-50% as fast as sf-dev. (There is also an intel skylake bmi2 compile I made) FYI, under a full AVX2 load CPUs get very hot so need good cooling to not downclock.

All testing done with my same 320 game no draw book on 3950x with 1 thread and full 6 man EGTB on nvme ssd. I used Brainfish compile with large pages because it is the fastest Stockfish on my system.

Code: Select all

15s+0.15s
Score of StockFiNN vs BrainFish_200420: 66 - 155 - 99 [0.361]
Elo difference: -99.2 +/- 32.4, LOS: 0.0 %, DrawRatio: 30.9 %

30s+0.3s
Score of StockFiNN vs BrainFish_200420: 72 - 127 - 121 [0.414]
Elo difference: -60.3 +/- 30.2, LOS: 0.0 %, DrawRatio: 37.8 %

60s+0.6s 
Score of StockFiNN vs BrainFish_200420: 74 - 103 - 143 [0.455]
Elo difference: -31.6 +/- 28.4, LOS: 1.5 %, DrawRatio: 44.7 %

120+1.2s 
Score of StockFiNN vs BrainFish_200420: 79 - 75 - 166 [0.506]
Elo difference: 4.3 +/- 26.4, LOS: 62.6 %, DrawRatio: 51.9 %

Code: Select all

fixed 100k nodes 
Score of StockFiNN vs BrainFish_200420: 144 - 95 - 81 [0.577]
Elo difference: 53.6 +/- 33.2, LOS: 99.9 %, DrawRatio: 25.3 %

fixed 1 million nodes 
Score of StockFiNN vs BrainFish_200420: 125 - 66 - 129 [0.592]
Elo difference: 64.8 +/- 29.6, LOS: 100.0 %, DrawRatio: 40.3 %

Testing at 10s+0.1s does not work well as there is too much variance and overhead with initializing NN search. But it seems to scale about 30 elo per doubling better than Stockfish!

[Milos]

This evaluation is created from scratch solely a 3 layer neural network that I started from a random initialization, so while it has search in common with Stockfish the evaluation is not a cloned and tweaked as well like a lot of SF clones that still have NN in the name, and merely mean it has some sort of learning function where it saves positions where it fails low/high. Like Leela, this does all its learning offline, not while playing matches.

Forked on my github from nodchips SF-NNUE repository, working towards symbiotic training with Leelenstein. You can download StockFiNN 0.1 and follow progress here: https://www.patreon.com/posts/stockfinn-0-1-38717611

It is progressing very fast so testing is in flux, but here are some scaling results worth seeing.
Using the default avx2 compile I posted I get about 1.6-1.7M nps with the default 1 thread bench, it's about 40-50% as fast as sf-dev. (There is also an intel skylake bmi2 compile I made) FYI, under a full AVX2 load CPUs get very hot so need good cooling to not downclock.

All testing done with my same 320 game no draw book on 3950x with 1 thread and full 6 man EGTB on nvme ssd. I used Brainfish compile with large pages because it is the fastest Stockfish on my system.

Code: Select all

15s+0.15s
Score of StockFiNN vs BrainFish_200420: 66 - 155 - 99 [0.361]
Elo difference: -99.2 +/- 32.4, LOS: 0.0 %, DrawRatio: 30.9 %

30s+0.3s
Score of StockFiNN vs BrainFish_200420: 72 - 127 - 121 [0.414]
Elo difference: -60.3 +/- 30.2, LOS: 0.0 %, DrawRatio: 37.8 %

60s+0.6s 
Score of StockFiNN vs BrainFish_200420: 74 - 103 - 143 [0.455]
Elo difference: -31.6 +/- 28.4, LOS: 1.5 %, DrawRatio: 44.7 %

120+1.2s 
Score of StockFiNN vs BrainFish_200420: 79 - 75 - 166 [0.506]
Elo difference: 4.3 +/- 26.4, LOS: 62.6 %, DrawRatio: 51.9 %

Code: Select all

fixed 100k nodes 
Score of StockFiNN vs BrainFish_200420: 144 - 95 - 81 [0.577]
Elo difference: 53.6 +/- 33.2, LOS: 99.9 %, DrawRatio: 25.3 %

fixed 1 million nodes 
Score of StockFiNN vs BrainFish_200420: 125 - 66 - 129 [0.592]
Elo difference: 64.8 +/- 29.6, LOS: 100.0 %, DrawRatio: 40.3 %

Testing at 10s+0.1s does not work well as there is too much variance and overhead with initializing NN search. But it seems to scale about 30 elo per doubling better than Stockfish!

What's the point of yet another fork of SF-NNUE? Any actual changes in NN or search to accommodate it, or TM?

[Raphexon]

This evaluation is created from scratch solely a 3 layer neural network that I started from a random initialization, so while it has search in common with Stockfish the evaluation is not a cloned and tweaked as well like a lot of SF clones that still have NN in the name, and merely mean it has some sort of learning function where it saves positions where it fails low/high. Like Leela, this does all its learning offline, not while playing matches.

Forked on my github from nodchips SF-NNUE repository, working towards symbiotic training with Leelenstein. You can download StockFiNN 0.1 and follow progress here: https://www.patreon.com/posts/stockfinn-0-1-38717611

It is progressing very fast so testing is in flux, but here are some scaling results worth seeing.
Using the default avx2 compile I posted I get about 1.6-1.7M nps with the default 1 thread bench, it's about 40-50% as fast as sf-dev. (There is also an intel skylake bmi2 compile I made) FYI, under a full AVX2 load CPUs get very hot so need good cooling to not downclock.

All testing done with my same 320 game no draw book on 3950x with 1 thread and full 6 man EGTB on nvme ssd. I used Brainfish compile with large pages because it is the fastest Stockfish on my system.

Code: Select all

15s+0.15s
Score of StockFiNN vs BrainFish_200420: 66 - 155 - 99 [0.361]
Elo difference: -99.2 +/- 32.4, LOS: 0.0 %, DrawRatio: 30.9 %

30s+0.3s
Score of StockFiNN vs BrainFish_200420: 72 - 127 - 121 [0.414]
Elo difference: -60.3 +/- 30.2, LOS: 0.0 %, DrawRatio: 37.8 %

60s+0.6s 
Score of StockFiNN vs BrainFish_200420: 74 - 103 - 143 [0.455]
Elo difference: -31.6 +/- 28.4, LOS: 1.5 %, DrawRatio: 44.7 %

120+1.2s 
Score of StockFiNN vs BrainFish_200420: 79 - 75 - 166 [0.506]
Elo difference: 4.3 +/- 26.4, LOS: 62.6 %, DrawRatio: 51.9 %

Code: Select all

fixed 100k nodes 
Score of StockFiNN vs BrainFish_200420: 144 - 95 - 81 [0.577]
Elo difference: 53.6 +/- 33.2, LOS: 99.9 %, DrawRatio: 25.3 %

fixed 1 million nodes 
Score of StockFiNN vs BrainFish_200420: 125 - 66 - 129 [0.592]
Elo difference: 64.8 +/- 29.6, LOS: 100.0 %, DrawRatio: 40.3 %

Testing at 10s+0.1s does not work well as there is too much variance and overhead with initializing NN search. But it seems to scale about 30 elo per doubling better than Stockfish!

What's the point of yet another fork of SF-NNUE? Any actual changes in NN or search to accommodate it, or TM?

Only difference is netsize, which is literally changing a single number in the source.

[lkaufman]

This evaluation is created from scratch solely a 3 layer neural network that I started from a random initialization, so while it has search in common with Stockfish the evaluation is not a cloned and tweaked as well like a lot of SF clones that still have NN in the name, and merely mean it has some sort of learning function where it saves positions where it fails low/high. Like Leela, this does all its learning offline, not while playing matches.

Forked on my github from nodchips SF-NNUE repository, working towards symbiotic training with Leelenstein. You can download StockFiNN 0.1 and follow progress here: https://www.patreon.com/posts/stockfinn-0-1-38717611

It is progressing very fast so testing is in flux, but here are some scaling results worth seeing.
Using the default avx2 compile I posted I get about 1.6-1.7M nps with the default 1 thread bench, it's about 40-50% as fast as sf-dev. (There is also an intel skylake bmi2 compile I made) FYI, under a full AVX2 load CPUs get very hot so need good cooling to not downclock.

All testing done with my same 320 game no draw book on 3950x with 1 thread and full 6 man EGTB on nvme ssd. I used Brainfish compile with large pages because it is the fastest Stockfish on my system.

Code: Select all

15s+0.15s
Score of StockFiNN vs BrainFish_200420: 66 - 155 - 99 [0.361]
Elo difference: -99.2 +/- 32.4, LOS: 0.0 %, DrawRatio: 30.9 %

30s+0.3s
Score of StockFiNN vs BrainFish_200420: 72 - 127 - 121 [0.414]
Elo difference: -60.3 +/- 30.2, LOS: 0.0 %, DrawRatio: 37.8 %

60s+0.6s 
Score of StockFiNN vs BrainFish_200420: 74 - 103 - 143 [0.455]
Elo difference: -31.6 +/- 28.4, LOS: 1.5 %, DrawRatio: 44.7 %

120+1.2s 
Score of StockFiNN vs BrainFish_200420: 79 - 75 - 166 [0.506]
Elo difference: 4.3 +/- 26.4, LOS: 62.6 %, DrawRatio: 51.9 %

Code: Select all

fixed 100k nodes 
Score of StockFiNN vs BrainFish_200420: 144 - 95 - 81 [0.577]
Elo difference: 53.6 +/- 33.2, LOS: 99.9 %, DrawRatio: 25.3 %

fixed 1 million nodes 
Score of StockFiNN vs BrainFish_200420: 125 - 66 - 129 [0.592]
Elo difference: 64.8 +/- 29.6, LOS: 100.0 %, DrawRatio: 40.3 %

Testing at 10s+0.1s does not work well as there is too much variance and overhead with initializing NN search. But it seems to scale about 30 elo per doubling better than Stockfish!

I know of opening books with low draw rates, but what is a "no draw book"?

[jjoshua2]

It's a low draw book I made for Leelenstein playing two variants with a full opening book and then taking out all the games that ended in draws. It ends up getting a nice split between draw, won, and decisive games that has worked for me to maximize elo spreads while still being representative. It might bias results for testing Leelenstein vs SF, but works great for testing engines against themselves or similar.

[jjoshua2]

What's the point of yet another fork of SF-NNUE? Any actual changes in NN or search to accommodate it, or TM?

The engine fork is not the interesting part. The interesting part is that I have trained a net that is as strong as Stockfish itself in a short TC 1 thread, and very rapidly without too many resources. And still gaining a lot

[Milos]

What's the point of yet another fork of SF-NNUE? Any actual changes in NN or search to accommodate it, or TM?

The engine fork is not the interesting part. The interesting part is that I have trained a net that is as strong as Stockfish itself in a short TC 1 thread, and very rapidly without too many resources. And still gaining a lot

I see, but net itself is only different to SF-NNUE one in size right?
So can actual binary accept different net sizes (like it's the case with Lc0)? Would be good to have a unique version of main binary that is up to date with SFdev and that accepts different nets.

[Chessqueen]

What's the point of yet another fork of SF-NNUE? Any actual changes in NN or search to accommodate it, or TM?

The engine fork is not the interesting part. The interesting part is that I have trained a net that is as strong as Stockfish itself in a short TC 1 thread, and very rapidly without too many resources. And still gaining a lot

So you are saying that this StockFINN might be the Next TCEC Champion even better than the latest Stockfish that beat LCO a few days ago

[Raphexon]

What's the point of yet another fork of SF-NNUE? Any actual changes in NN or search to accommodate it, or TM?

The engine fork is not the interesting part. The interesting part is that I have trained a net that is as strong as Stockfish itself in a short TC 1 thread, and very rapidly without too many resources. And still gaining a lot

How much data did you even need in the end?
At least I'm glad that somebody you have surpassed my 26-6 and 27-6 nets now in little time.

Hope I can get my stumbles with the latest binaries quickly fixed so I can upgrade mine too.

[jjoshua2]

Important point I forget to state before was that all my tests were using slowmover 60 and defaults of course for Stockfish. My last net had about 950 million positions and trained for maybe 5 billion steps/positions (can't remember)? I'm training my next net a lot slower, almost 7 billion to first learning rate drop.