StockFiNN Release and Scaling

[brianr]

The funniest is when someone comes and complains that they need it to cover electricity cost . Or that they did some tremendous work on network training that a monkey could do, just because they have some nice database of games that they collected as a hobby and now decided to gain some coin on it.
Never underestimate naivety of ppl.

I have been training Leela nets for about two years, and a few Giraffe nets before that. You may be significantly underestimating what is involved. It is not trivial to learn how to train a net. Once you have some idea of how to do it, setting up all of the various libraries and tools is very exacting. In addition, massive amounts of input data must be prepared, and the results must be tested in matches. Figure several months of work. Yes, at that point a net of about 3,000 Elo can be trained in a few days. However, it does get increasingly harder to improve a strong net. A strong, but not top-tier net can easily take several weeks of training time. The very best nets represent many months of work.

[Milos]

The funniest is when someone comes and complains that they need it to cover electricity cost . Or that they did some tremendous work on network training that a monkey could do, just because they have some nice database of games that they collected as a hobby and now decided to gain some coin on it.
Never underestimate naivety of ppl.

I have been training Leela nets for about two years, and a few Giraffe nets before that. You may be significantly underestimating what is involved. It is not trivial to learn how to train a net. Once you have some idea of how to do it, setting up all of the various libraries and tools is very exacting. In addition, massive amounts of input data must be prepared, and the results must be tested in matches. Figure several months of work. Yes, at that point a net of about 3,000 Elo can be trained in a few days. However, it does get increasingly harder to improve a strong net. A strong, but not top-tier net can easily take several weeks of training time. The very best nets represent many months of work.

I do machine learning for a living, so when I refereed to "any monkey can train" I meant any intern we get for a few months can do it more, less successfully. I understand something like this is quite a task for an "enthusiast". But for someone that is quite familiar with those things, most of arguments sound like a bad excuse. Only thing one needs for data preparation is to be versed in writing python/perl/or any kind of PGN processing scripts. This is not some rocket science, all data is already (trivially) annotated so the biggest obstacle from real ML scientific world is a moot point.
The whole hyper-parameter set is reduced to net size, LR and maybe some temperature parameter. In my world that is rather trivial.

All to simplify things. The knowledge threshold one needs to write 3000 Elo engine from scratch and to train NN with existing engine and training binaries to 3000 Elo is worlds apart.

Regarding strongest network, if NN chess was really interesting to ML community once you put it on kaggle I bet you'd get in couple of months much stronger net than any Leelastein of whatever current individual effort is.

[brianr]

And a professional will do that for $600 bucks a month!

[Milos]

And a professional will do that for $600 bucks a month!

Most of ML research is done by graduate and even master students that are paid (if they are even paid anything) less than that .

[dkappe]

Most of ML research is done by graduate and even master students that are paid (if they are even paid anything) less than that .

I used to make my living developing medical device software, including machine learning components. I used to joke that at least 50% of the the money we made was from undoing the damage the graduate students had done (usually in the form of matlab or tensorflow spaghetti code).

You may have a different opinion, but I’d say data prep and training on games like chess and go is a bit more difficult than for your garden variety deep learning effort, especially since the resulting network drives a mcts search with complex behavior of its own.

[Milos]

I used to make my living developing medical device software, including machine learning components. I used to joke that at least 50% of the the money we made was from undoing the damage the graduate students had done (usually in the form of matlab or tensorflow spaghetti code).

You may have a different opinion, but I’d say data prep and training on games like chess and go is a bit more difficult than for your garden variety deep learning effort, especially since the resulting network drives a mcts search with complex behavior of its own.

That's why one should never put a graduate student to develop any kind of (serious) commercial product .
Jokes aside you are quite right there.

Regarding training of Leela, I understand that things are quite more complex than typical kind of deep learning/NLP effort done today, but difficulty comes mainly from a lacking (and difficult to properly implement) testing methodology.
With SFNNUE that is really not the case, testing methodology exists and NN itself is quite an independent component. I predict that once SF main developers adopt NN approach to eval and start rigorously testing (and training NNs) that this semi-commercial enthusiast aspect is going to disappear.

[Gejsi Marku]

Another test: 15min+15sec TC
So far after playing about 34 games each with a random 3 move (6plies) book.

Engine /Score St Le St /S-B
1: Stockfinn /18.5/34 ················· 01=========1===== =====0=11=====101 /290.50
2: Leelenstein 15 /17.5/33 10=========0===== ················· 11=======101==== /281.00
3: Stockfish_Jun29_20_x64_bmi2 /14.0/33 =====1=00=====010 00=======010==== ················· /252.50

[dkappe]

difficulty comes mainly from a lacking (and difficult to properly implement) testing methodology.

Due to the enormous amounts of computation necessary to train and test a series of leela nets, tests unfold over months, not minutes or hours. That, rather than the lack of a good method, is what hampers that project. It looks like testing and training are much shorter for NNUE, which may be to its advantage.

[Raphexon]

difficulty comes mainly from a lacking (and difficult to properly implement) testing methodology.

Due to the enormous amounts of computation necessary to train and test a series of leela nets, tests unfold over months, not minutes or hours. That, rather than the lack of a good method, is what hampers that project. It looks like testing and training are much shorter for NNUE, which may be to its advantage.

Is that actually the case, or are we simply getting away with it?

[Milos]

difficulty comes mainly from a lacking (and difficult to properly implement) testing methodology.

Due to the enormous amounts of computation necessary to train and test a series of leela nets, tests unfold over months, not minutes or hours. That, rather than the lack of a good method, is what hampers that project. It looks like testing and training are much shorter for NNUE, which may be to its advantage.

Is that actually the case, or are we simply getting away with it?

Once SF-NNUE is introduced into Stockfish testing framework it will simplify things for sure.