NNUE - only from own engine?

[ChickenLogic]

...

And again it shows that people here talking about NNUE know very little about how much it takes to do properly. Generating your own data that is just about half way decent and confirming it as such takes Stockfish well over a couple of weeks, and that is with multiple V100s and threadrippers. You also need multiple full training runs to confirm your new method isn't a fluke. If you don't happen to have multiple PCs in the range of multiple thousand of dollars there is no way you produce anything close to top engines without taking 'their' data. There are people in the Stockfish project that exclusively train neural nets. You sir just see that Ethereal and Halogen progress quickly but fail to see the insane amount of hardware they have in the background aside from open bench. OpenBench is also 'sponsored' by noob. You really think a single engine author can compete with a guy who has his own data center? And even if you join openbench, you still need insane amounts of hardware for training on your own.

I guess you didn't read my post very carefully.
I've generated about 2B d8 self play games which I use for training Seer on a R5 3600. I train solely using a GTX 950. Seer is a top 15 engine.
Koivisto is trained on 500M mixed depth self play games generated using comparatively modest hardware. Koivisto's training code is CPU based. Koivisto is a top 10 engine (at least).
BitGenie is a 3100 elo engine. It's selfplay data was generated on 4 thread laptop and it was also trained on said laptop.

So no, it doesn't take weeks of compute with "multiple V100s and threadrippers" to obtain competitive results. Maybe that's what it would take to beat Stockfish, but that's not what I claimed.

We just have a very different understanding of 'competitive'. According to CCRL, only 2 engines below #15 use NNs and only one of them uses NNUE (to my knowledge). These would be Winter (using classical eval and CNN iirc) and Halogen. So, Seer is not really competitive with NNUE engines but still easily beats purely classical engines. That NNUE is superior to HCE shouldn't be a question. I was one of the first to show that. So yes, if we consider all engines, Seer is competitive.
For me HCE is dead - I don't see them as part of the competition at all. Either way, I'm glad that there are other ways than the "SF way" to surpass HCE with 'moderate amounts of hardware juice'. I don't mean to discourage you or belittle your work given that you wrote your own trainer and also generate your own data. This is very commendable - especially given how many people don't do either. So keep on going

(I'm sorry I didn't specify what I meant by 'doing [NNUE training] properly'. I am aware that this could be seen as shifting goalposts. As silly as it sounds, sometimes I just forget that reading someone's mind is not a thing, especially over the internet)

[Madeleine Birchfield]

According to CCRL, only 2 engines below #15 use NNs and only one of them uses NNUE (to my knowledge). These would be Winter (using classical eval and CNN iirc) and Halogen.

Fizbo 2 also uses a neural network.

[amanjpro]

...

And again it shows that people here talking about NNUE know very little about how much it takes to do properly. Generating your own data that is just about half way decent and confirming it as such takes Stockfish well over a couple of weeks, and that is with multiple V100s and threadrippers. You also need multiple full training runs to confirm your new method isn't a fluke. If you don't happen to have multiple PCs in the range of multiple thousand of dollars there is no way you produce anything close to top engines without taking 'their' data. There are people in the Stockfish project that exclusively train neural nets. You sir just see that Ethereal and Halogen progress quickly but fail to see the insane amount of hardware they have in the background aside from open bench. OpenBench is also 'sponsored' by noob. You really think a single engine author can compete with a guy who has his own data center? And even if you join openbench, you still need insane amounts of hardware for training on your own.

I guess you didn't read my post very carefully.
I've generated about 2B d8 self play games which I use for training Seer on a R5 3600. I train solely using a GTX 950. Seer is a top 15 engine.
Koivisto is trained on 500M mixed depth self play games generated using comparatively modest hardware. Koivisto's training code is CPU based. Koivisto is a top 10 engine (at least).
BitGenie is a 3100 elo engine. It's selfplay data was generated on 4 thread laptop and it was also trained on said laptop.

So no, it doesn't take weeks of compute with "multiple V100s and threadrippers" to obtain competitive results. Maybe that's what it would take to beat Stockfish, but that's not what I claimed.

We just have a very different understanding of 'competitive'. According to CCRL, only 2 engines below #15 use NNs and only one of them uses NNUE (to my knowledge). These would be Winter (using classical eval and CNN iirc) and Halogen. So, Seer is not really competitive with NNUE engines but still easily beats purely classical engines. That NNUE is superior to HCE shouldn't be a question. I was one of the first to show that. So yes, if we consider all engines, Seer is competitive.
For me HCE is dead - I don't see them as part of the competition at all. Either way, I'm glad that there are other ways than the "SF way" to surpass HCE with 'moderate amounts of hardware juice'. I don't mean to discourage you or belittle your work given that you wrote your own trainer and also generate your own data. This is very commendable - especially given how many people don't do either. So keep on going

(I'm sorry I didn't specify what I meant by 'doing [NNUE training] properly'. I am aware that this could be seen as shifting goalposts. As silly as it sounds, sometimes I just forget that reading someone's mind is not a thing, especially over the internet)

Dude some research doesn't hurt.... Zahak, BitGenie, Velvet, Vajolet and many more do NNUE, can you also redefine NNUE so your statement holds?

[connor_mcmonigle]

...

And again it shows that people here talking about NNUE know very little about how much it takes to do properly. Generating your own data that is just about half way decent and confirming it as such takes Stockfish well over a couple of weeks, and that is with multiple V100s and threadrippers. You also need multiple full training runs to confirm your new method isn't a fluke. If you don't happen to have multiple PCs in the range of multiple thousand of dollars there is no way you produce anything close to top engines without taking 'their' data. There are people in the Stockfish project that exclusively train neural nets. You sir just see that Ethereal and Halogen progress quickly but fail to see the insane amount of hardware they have in the background aside from open bench. OpenBench is also 'sponsored' by noob. You really think a single engine author can compete with a guy who has his own data center? And even if you join openbench, you still need insane amounts of hardware for training on your own.

I guess you didn't read my post very carefully.
I've generated about 2B d8 self play games which I use for training Seer on a R5 3600. I train solely using a GTX 950. Seer is a top 15 engine.
Koivisto is trained on 500M mixed depth self play games generated using comparatively modest hardware. Koivisto's training code is CPU based. Koivisto is a top 10 engine (at least).
BitGenie is a 3100 elo engine. It's selfplay data was generated on 4 thread laptop and it was also trained on said laptop.

So no, it doesn't take weeks of compute with "multiple V100s and threadrippers" to obtain competitive results. Maybe that's what it would take to beat Stockfish, but that's not what I claimed.

We just have a very different understanding of 'competitive'. According to CCRL, only 2 engines below #15 use NNs and only one of them uses NNUE (to my knowledge). These would be Winter (using classical eval and CNN iirc) and Halogen. So, Seer is not really competitive with NNUE engines but still easily beats purely classical engines. That NNUE is superior to HCE shouldn't be a question. I was one of the first to show that. So yes, if we consider all engines, Seer is competitive.
For me HCE is dead - I don't see them as part of the competition at all. Either way, I'm glad that there are other ways than the "SF way" to surpass HCE with 'moderate amounts of hardware juice'. I don't mean to discourage you or belittle your work given that you wrote your own trainer and also generate your own data. This is very commendable - especially given how many people don't do either. So keep on going

Thanks. I guess we have to clarify what's meant by NNUE when making these comparisons. Seer uses an incrementally updated neural network as does Halogen, Winter, Slowchess, Berserk and Koivisto. All these engines use a fairly different network topology as compared to Stockfish (or any prior version of Stockfish) and don't rely on SF inference code/training code. The latest version of Seer is actually a fair bit stronger than Igel and Nemorino in my testing as well. I'd consider that 'competitive'. In fact, anything over 3100 elo is sufficiently competitive for TCEC (which was the original context in which I was using the term "competitive").

[dkappe]

According to CCRL, only 2 engines below #15 use NNs and only one of them uses NNUE (to my knowledge). These would be Winter (using classical eval and CNN iirc) and Halogen.

Fizbo 2 also uses a neural network.

Dark Toga and Toga III. I think most engines over 3100 CCRL have a good chance of having an NNUE inside.

[connor_mcmonigle]

...

Dude some research doesn't hurt.... Zahak, BitGenie, Velvet, Vajolet and many more do NNUE, can you also redefine NNUE so your statement holds?

+1

[ChickenLogic]

...

And again it shows that people here talking about NNUE know very little about how much it takes to do properly. Generating your own data that is just about half way decent and confirming it as such takes Stockfish well over a couple of weeks, and that is with multiple V100s and threadrippers. You also need multiple full training runs to confirm your new method isn't a fluke. If you don't happen to have multiple PCs in the range of multiple thousand of dollars there is no way you produce anything close to top engines without taking 'their' data. There are people in the Stockfish project that exclusively train neural nets. You sir just see that Ethereal and Halogen progress quickly but fail to see the insane amount of hardware they have in the background aside from open bench. OpenBench is also 'sponsored' by noob. You really think a single engine author can compete with a guy who has his own data center? And even if you join openbench, you still need insane amounts of hardware for training on your own.

I guess you didn't read my post very carefully.
I've generated about 2B d8 self play games which I use for training Seer on a R5 3600. I train solely using a GTX 950. Seer is a top 15 engine.
Koivisto is trained on 500M mixed depth self play games generated using comparatively modest hardware. Koivisto's training code is CPU based. Koivisto is a top 10 engine (at least).
BitGenie is a 3100 elo engine. It's selfplay data was generated on 4 thread laptop and it was also trained on said laptop.

So no, it doesn't take weeks of compute with "multiple V100s and threadrippers" to obtain competitive results. Maybe that's what it would take to beat Stockfish, but that's not what I claimed.

We just have a very different understanding of 'competitive'. According to CCRL, only 2 engines below #15 use NNs and only one of them uses NNUE (to my knowledge). These would be Winter (using classical eval and CNN iirc) and Halogen. So, Seer is not really competitive with NNUE engines but still easily beats purely classical engines. That NNUE is superior to HCE shouldn't be a question. I was one of the first to show that. So yes, if we consider all engines, Seer is competitive.
For me HCE is dead - I don't see them as part of the competition at all. Either way, I'm glad that there are other ways than the "SF way" to surpass HCE with 'moderate amounts of hardware juice'. I don't mean to discourage you or belittle your work given that you wrote your own trainer and also generate your own data. This is very commendable - especially given how many people don't do either. So keep on going

(I'm sorry I didn't specify what I meant by 'doing [NNUE training] properly'. I am aware that this could be seen as shifting goalposts. As silly as it sounds, sometimes I just forget that reading someone's mind is not a thing, especially over the internet)

Dude some research doesn't hurt.... Zahak, BitGenie, Velvet, Vajolet and many more do NNUE, can you also redefine NNUE so your statement holds?

Do you know what "to my knowledge" means? I never claimed to know all engines. I should've known about vajolet though. So, I correct my statement from "Seer is close to the bottom of nnue engines" to "Seer is in the middle of the pack considering only NN/NNUE driven engines". Happy now? The point that getting to the top of NNUE engines without any outside help while being a single dev would be nothing short of a miracle remains.
Seer was one of the first NNUE engines and naturally had time to create a strong trainer. Saying "just use 500m d8 fens" is not what happened in reality. Well over a year of dev time went into that which was another point I made about NNUE taking time to do properly from scratch.

The fact that Seer can just beat HCE engines with 500m fens confirms that there was a lot of testing. And it certainly did take more than a few months to get there.

[Rebel]

I believe most of the major rating list testers (CEGT, CCRL and others) are not interested in SF NNUE unless the net is trained solely on the engine's own games

That's true.

I think the issue is important enough open a discussion.

NNUE eval is the result from :

1. Training software (freely available)
2. Quality of the EPD's (hard own work)
3. NNUE implementation (freely available)

Since the elo is in the EPD (and is the gold digging part) I see no good reason reason to put a limitation on the creative part (the quality of the EPD's). Further it is discrimination to starters who are forced to write a good HCE eval first. Everybody should be free to create his own EPD database as he pleases, train it as he pleases, implement it as he pleases.

I see only one limitation, using an existing NNUE from someone else. It means the creative (and hard) part is skipped. Fire comes to mind, I don't test it.

Lately, nearly all of the progress for SF nets have been made in the trainer - not the data. What you call the 'EPD' part (besides the only EPDs used are the opening books for data generation, EPD itself is not suitable for training) can be a gold mine. I've shown that with converted Leela data. However, you severely underestimate the work that goes into the trainer and into testing the trainer. Even if you were to build on top of SF's trainer, there are lot's of things to do differently and a lot of things to improve. It is just as much of a 'gold mine' as the training data itself.

I do not underestimate the work on the trainer or the work on different net structures. Note the subject line.

As for fire, 'his' implementation wasn't new. He didn't even bother to train a net on his own; he straight up took it from Sergio's series of nets specifically trained for SF on SF data with SF's trainer. That's the highest level of lazy one can 'achieve' really.

[MartinBryant]

What tournaments, testers, etc. want to see from engines is their doing something at least somewhat unique with respect to their evaluation functions.

Yes. I remember Ted, one of the testers, bemoaning a few weeks ago that everything tasted of fish

A question for the net experts then...

Given that at the moment there are still several flavours of net, generated in different ways using different data, as time goes on and they all improve, given enough time, won't they all converge towards a point of 'perfect' strategic play and agree on every positional evaluation?
Then they won't all taste of 'fish' but of 'perfection'?
Early days yet but could nets (eventually) be the death of computer chess?

[amanjpro]

What tournaments, testers, etc. want to see from engines is their doing something at least somewhat unique with respect to their evaluation functions.

Yes. I remember Ted, one of the testers, bemoaning a few weeks ago that everything tasted of fish

A question for the net experts then...

Given that at the moment there are still several flavours of net, generated in different ways using different data, as time goes on and they all improve, given enough time, won't they all converge towards a point of 'perfect' strategic play and agree on every positional evaluation?
Then they won't all taste of 'fish' but of 'perfection'?
Early days yet but could nets (eventually) be the death of computer chess?

What is best for SF can't work for Zahak, or others... I don't have a data center to train my net, so I go for something less perfect (in the sense of sheer strength), but more perfect in the sense of trainability on my hardware