SF-NN embedded

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

User avatar
Deberger
Posts: 91
Joined: Sat Nov 02, 2019 6:42 pm
Full name: ɹǝƃɹǝqǝᗡ ǝɔnɹꓭ

Re: SF-NN embedded

Post by Deberger »

Dann Corbit wrote: Sat Aug 29, 2020 6:24 am I think the embedded file is misguided.
For instance, I build a bunch of different binaries, and shall each one have a 20MB pile of goo sitting in it's data space?
Imagine if all of the stockfish variants follow suit.
Then we will have 3-4x 20MB pile multiplied by variant count.
I think it would make me cry,.

There is a reason we don't embed 5 man tablebase file sets in the binaries.
If it were that hard to browse to a file, nobody could use a tablebase.

Please, for the love of all that is good and true, no more embedding of 20MB files.

I move files across the network like mad. The volume of the file makes the virus scan time go up something like quadratically. A 20 MB file causes a big delay. A 60 MB file causes an atrocious delay. I do a build for each new variant of each new engine. We are talking about serious problems here.

Please, please, please. I am begging down on my knees. No embedding of TB files or Networks.
It's already done:

https://github.com/official-stockfish/S ... c8f127a741
Joerg Oster
Posts: 937
Joined: Fri Mar 10, 2006 4:29 pm
Location: Germany

Re: SF-NN embedded

Post by Joerg Oster »

Deberger wrote: Sat Aug 29, 2020 10:37 pm
Dann Corbit wrote: Sat Aug 29, 2020 6:24 am I think the embedded file is misguided.
For instance, I build a bunch of different binaries, and shall each one have a 20MB pile of goo sitting in it's data space?
Imagine if all of the stockfish variants follow suit.
Then we will have 3-4x 20MB pile multiplied by variant count.
I think it would make me cry,.

There is a reason we don't embed 5 man tablebase file sets in the binaries.
If it were that hard to browse to a file, nobody could use a tablebase.

Please, for the love of all that is good and true, no more embedding of 20MB files.

I move files across the network like mad. The volume of the file makes the virus scan time go up something like quadratically. A 20 MB file causes a big delay. A 60 MB file causes an atrocious delay. I do a build for each new variant of each new engine. We are talking about serious problems here.

Please, please, please. I am begging down on my knees. No embedding of TB files or Networks.
It's already done:

https://github.com/official-stockfish/S ... c8f127a741
I can't believe it ...
Jörg Oster
Dann Corbit
Posts: 12538
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: SF-NN embedded

Post by Dann Corbit »

I hope that there are some simple instructions for removal.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
syzygy
Posts: 5557
Joined: Tue Feb 28, 2012 11:56 pm

Re: SF-NN embedded

Post by syzygy »

Joerg Oster wrote: Sat Aug 29, 2020 11:14 pm I can't believe it ...
I think it is useful for official releases. There are probably many users that download Stockfish at most once a year and would have a lot of trouble to configure the network file correctly.

For developers and advanced users, embedded network files are just a nuisance.
Alayan
Posts: 550
Joined: Tue Nov 19, 2019 8:48 pm
Full name: Alayan Feh

Re: SF-NN embedded

Post by Alayan »

Storing dozens of different Stockfish versions that share the same main net and have only minor differences between them or just some search changes is a niche use case.
kk
Posts: 21
Joined: Wed Nov 23, 2016 8:23 am
Location: UK

Re: SF-NN embedded

Post by kk »

The solution to an embedded net appears simple to me, for those against embedding. I can understand the confusion of a once a year download user where an embedded net would be easy to install, so in that situation I support and applaud the idea of an embedded net file. For those who understand how to point (set the path) to an external netfile why not have a checkbox in the param file labelled something like 'use embedded net'. Ticked = use embedded, unticked = use external net.
It is an exciting time to be able to play around with netfiles to determine how the program reacts with different files.
Naturally the 'annual download merchant' would be happy to have a working program without the netfile hassle and the experimenter can tweak to his hearts desire.
corres
Posts: 3657
Joined: Wed Nov 18, 2015 11:41 am
Location: hungary

Re: SF-NN embedded

Post by corres »

syzygy wrote: Sun Aug 30, 2020 4:21 am ...
I think it is useful for official releases. There are probably many users that download Stockfish at most once a year and would have a lot of trouble to configure the network file correctly.
For developers and advanced users, embedded network files are just a nuisance.
I agree.
As we can see on abrok.eu/stockfish site nowadays the newer and newer Stoclfish have only small difference in code and in Elo. So if somebody use for e.g. 20200806 dated Stockfish with the most stronger NNUE-net
(nn-82215d0fd0df-814.bin) he get near the same Stockfish as he use the last "embedded" version.
Note
1, I do not like if a program is bounded to the Net, especially in the phase of compilation.
Net is a dangerous place for every program and I want to defense my machine and my own Stockfish from any danger on the Net.
2, Developer of Stockfish said they made a "simplified" Make - file for Stockfish, but they forgot to mention which Windows compiler is good for the new Stockfish and its "simple" Make- file.
For the "Classical" Stockfish is good the Mingw-gcc x86-64 compiler, but who know what is good for NON-EMBEDDED! SF+NNUE - version?
zullil
Posts: 6442
Joined: Tue Jan 09, 2007 12:31 am
Location: PA USA
Full name: Louis Zulli

Re: SF-NN embedded

Post by zullil »

Dann Corbit wrote: Sun Aug 30, 2020 12:43 am I hope that there are some simple instructions for removal.
"It is possible to disable embedding, and instead specify a specific directory, e.g. linux distros might want to use
CXXFLAGS="-DNNUE_EMBEDDING_OFF -DDEFAULT_NNUE_DIRECTORY=/usr/share/games/stockfish/" make -j ARCH=x86-64 profile-build"

Indeed, the following command invoked from the Stockfish/src directory will download the default net (if not already present) to that directory and build a stockfish binary free from having an embedded net. By default, that binary will use the net that was just download, and will find it without issue.

Code: Select all

CXXFLAGS="-DNNUE_EMBEDDING_OFF" make -j ARCH=x86-64-bmi2 profile-build
Vinvin
Posts: 5228
Joined: Thu Mar 09, 2006 9:40 am
Full name: Vincent Lejeune

Re: SF-NN embedded

Post by Vinvin »

Joerg Oster wrote: Sat Aug 29, 2020 11:14 pm
Deberger wrote: Sat Aug 29, 2020 10:37 pm
Dann Corbit wrote: Sat Aug 29, 2020 6:24 am I think the embedded file is misguided.
For instance, I build a bunch of different binaries, and shall each one have a 20MB pile of goo sitting in it's data space?
Imagine if all of the stockfish variants follow suit.
Then we will have 3-4x 20MB pile multiplied by variant count.
I think it would make me cry,.

There is a reason we don't embed 5 man tablebase file sets in the binaries.
If it were that hard to browse to a file, nobody could use a tablebase.

Please, for the love of all that is good and true, no more embedding of 20MB files.

I move files across the network like mad. The volume of the file makes the virus scan time go up something like quadratically. A 20 MB file causes a big delay. A 60 MB file causes an atrocious delay. I do a build for each new variant of each new engine. We are talking about serious problems here.

Please, please, please. I am begging down on my knees. No embedding of TB files or Networks.
It's already done:

https://github.com/official-stockfish/S ... c8f127a741
I can't believe it ...
Stockfish with embedded NN-file has no fewer functionalities than before.
And it has one more function to help low level users.
Alayan
Posts: 550
Joined: Tue Nov 19, 2019 8:48 pm
Full name: Alayan Feh

Re: SF-NN embedded

Post by Alayan »

It seems 20MB of disk space is prohibitively expensive, or that 20MB of download is incredibly slow on the dial-up internet of the complainers.