SF-NN embedded

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Deberger
Posts: 71
Joined: Sat Nov 02, 2019 5:42 pm
Full name: ɹǝƃɹǝqǝᗡ ǝɔnɹꓭ

Re: SF-NN embedded

Post by Deberger » Sat Aug 29, 2020 8:37 pm

Dann Corbit wrote:
Sat Aug 29, 2020 4:24 am
I think the embedded file is misguided.
For instance, I build a bunch of different binaries, and shall each one have a 20MB pile of goo sitting in it's data space?
Imagine if all of the stockfish variants follow suit.
Then we will have 3-4x 20MB pile multiplied by variant count.
I think it would make me cry,.

There is a reason we don't embed 5 man tablebase file sets in the binaries.
If it were that hard to browse to a file, nobody could use a tablebase.

Please, for the love of all that is good and true, no more embedding of 20MB files.

I move files across the network like mad. The volume of the file makes the virus scan time go up something like quadratically. A 20 MB file causes a big delay. A 60 MB file causes an atrocious delay. I do a build for each new variant of each new engine. We are talking about serious problems here.

Please, please, please. I am begging down on my knees. No embedding of TB files or Networks.
It's already done:

https://github.com/official-stockfish/S ... c8f127a741

Joerg Oster
Posts: 811
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: SF-NN embedded

Post by Joerg Oster » Sat Aug 29, 2020 9:14 pm

Deberger wrote:
Sat Aug 29, 2020 8:37 pm
Dann Corbit wrote:
Sat Aug 29, 2020 4:24 am
I think the embedded file is misguided.
For instance, I build a bunch of different binaries, and shall each one have a 20MB pile of goo sitting in it's data space?
Imagine if all of the stockfish variants follow suit.
Then we will have 3-4x 20MB pile multiplied by variant count.
I think it would make me cry,.

There is a reason we don't embed 5 man tablebase file sets in the binaries.
If it were that hard to browse to a file, nobody could use a tablebase.

Please, for the love of all that is good and true, no more embedding of 20MB files.

I move files across the network like mad. The volume of the file makes the virus scan time go up something like quadratically. A 20 MB file causes a big delay. A 60 MB file causes an atrocious delay. I do a build for each new variant of each new engine. We are talking about serious problems here.

Please, please, please. I am begging down on my knees. No embedding of TB files or Networks.
It's already done:

https://github.com/official-stockfish/S ... c8f127a741
I can't believe it ...
Jörg Oster

Dann Corbit
Posts: 11925
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: SF-NN embedded

Post by Dann Corbit » Sat Aug 29, 2020 10:43 pm

I hope that there are some simple instructions for removal.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

syzygy
Posts: 4891
Joined: Tue Feb 28, 2012 10:56 pm

Re: SF-NN embedded

Post by syzygy » Sun Aug 30, 2020 2:21 am

Joerg Oster wrote:
Sat Aug 29, 2020 9:14 pm
I can't believe it ...
I think it is useful for official releases. There are probably many users that download Stockfish at most once a year and would have a lot of trouble to configure the network file correctly.

For developers and advanced users, embedded network files are just a nuisance.

Alayan
Posts: 500
Joined: Tue Nov 19, 2019 7:48 pm
Full name: Alayan Feh

Re: SF-NN embedded

Post by Alayan » Sun Aug 30, 2020 2:28 am

Storing dozens of different Stockfish versions that share the same main net and have only minor differences between them or just some search changes is a niche use case.

kk
Posts: 20
Joined: Wed Nov 23, 2016 7:23 am
Location: UK

Re: SF-NN embedded

Post by kk » Sun Aug 30, 2020 9:55 am

The solution to an embedded net appears simple to me, for those against embedding. I can understand the confusion of a once a year download user where an embedded net would be easy to install, so in that situation I support and applaud the idea of an embedded net file. For those who understand how to point (set the path) to an external netfile why not have a checkbox in the param file labelled something like 'use embedded net'. Ticked = use embedded, unticked = use external net.
It is an exciting time to be able to play around with netfiles to determine how the program reacts with different files.
Naturally the 'annual download merchant' would be happy to have a working program without the netfile hassle and the experimenter can tweak to his hearts desire.

corres
Posts: 3657
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: SF-NN embedded

Post by corres » Sun Aug 30, 2020 10:22 am

syzygy wrote:
Sun Aug 30, 2020 2:21 am
...
I think it is useful for official releases. There are probably many users that download Stockfish at most once a year and would have a lot of trouble to configure the network file correctly.
For developers and advanced users, embedded network files are just a nuisance.
I agree.
As we can see on abrok.eu/stockfish site nowadays the newer and newer Stoclfish have only small difference in code and in Elo. So if somebody use for e.g. 20200806 dated Stockfish with the most stronger NNUE-net
(nn-82215d0fd0df-814.bin) he get near the same Stockfish as he use the last "embedded" version.
Note
1, I do not like if a program is bounded to the Net, especially in the phase of compilation.
Net is a dangerous place for every program and I want to defense my machine and my own Stockfish from any danger on the Net.
2, Developer of Stockfish said they made a "simplified" Make - file for Stockfish, but they forgot to mention which Windows compiler is good for the new Stockfish and its "simple" Make- file.
For the "Classical" Stockfish is good the Mingw-gcc x86-64 compiler, but who know what is good for NON-EMBEDDED! SF+NNUE - version?

zullil
Posts: 6442
Joined: Mon Jan 08, 2007 11:31 pm
Location: PA USA
Full name: Louis Zulli

Re: SF-NN embedded

Post by zullil » Sun Aug 30, 2020 10:26 am

Dann Corbit wrote:
Sat Aug 29, 2020 10:43 pm
I hope that there are some simple instructions for removal.
"It is possible to disable embedding, and instead specify a specific directory, e.g. linux distros might want to use
CXXFLAGS="-DNNUE_EMBEDDING_OFF -DDEFAULT_NNUE_DIRECTORY=/usr/share/games/stockfish/" make -j ARCH=x86-64 profile-build"

Indeed, the following command invoked from the Stockfish/src directory will download the default net (if not already present) to that directory and build a stockfish binary free from having an embedded net. By default, that binary will use the net that was just download, and will find it without issue.

Code: Select all

CXXFLAGS="-DNNUE_EMBEDDING_OFF" make -j ARCH=x86-64-bmi2 profile-build

Vinvin
Posts: 4864
Joined: Thu Mar 09, 2006 8:40 am
Full name: Vincent Lejeune

Re: SF-NN embedded

Post by Vinvin » Sun Aug 30, 2020 1:54 pm

Joerg Oster wrote:
Sat Aug 29, 2020 9:14 pm
Deberger wrote:
Sat Aug 29, 2020 8:37 pm
Dann Corbit wrote:
Sat Aug 29, 2020 4:24 am
I think the embedded file is misguided.
For instance, I build a bunch of different binaries, and shall each one have a 20MB pile of goo sitting in it's data space?
Imagine if all of the stockfish variants follow suit.
Then we will have 3-4x 20MB pile multiplied by variant count.
I think it would make me cry,.

There is a reason we don't embed 5 man tablebase file sets in the binaries.
If it were that hard to browse to a file, nobody could use a tablebase.

Please, for the love of all that is good and true, no more embedding of 20MB files.

I move files across the network like mad. The volume of the file makes the virus scan time go up something like quadratically. A 20 MB file causes a big delay. A 60 MB file causes an atrocious delay. I do a build for each new variant of each new engine. We are talking about serious problems here.

Please, please, please. I am begging down on my knees. No embedding of TB files or Networks.
It's already done:

https://github.com/official-stockfish/S ... c8f127a741
I can't believe it ...
Stockfish with embedded NN-file has no fewer functionalities than before.
And it has one more function to help low level users.

Alayan
Posts: 500
Joined: Tue Nov 19, 2019 7:48 pm
Full name: Alayan Feh

Re: SF-NN embedded

Post by Alayan » Sun Aug 30, 2020 2:12 pm

It seems 20MB of disk space is prohibitively expensive, or that 20MB of download is incredibly slow on the dial-up internet of the complainers.

Post Reply