NNUE architecture and training

[FlavusSnow]

Recently I've done a lot of training of various architectures. I train at quarter, half, and full data in order to get a learning curve for each architecture. I then run three comparison matches with an anchor version. The three matches are 1. puresearch, fixed depth (no pruning and no extensions) 2. Fixed node and 3. timed game. I do this to see what is best at ranking moves outright, whats best within the search parameters, and whats best when time matters. These three points, combined with the learning curve, allows me to evaluate the architecture and its potential to take on more data. I use it to forecast what shapes to train after later datagen rounds.

Here is the rub and I'm guessing this isnt specific to my setup or FunComp or anything like that:

Everything is plateauing and doing so clearly. My last round of training was on ~1.4B positions but even the best learning curve only forecasts another 15 Elo on the next doubling of data. Yet when I run gauntlets and analyze losses, its clear that FunComp is being out-played on evaluation. I do know that some of the larger, more complicated architectures don't come to life until a certain minimum of training data is obtained and I hold out hope that I'll see some of those start to hit as my data pool grows, so maybe there is that.

The question is, what tips would you give for self-training to break through this pattern? I suspect this is why a lot of engines train on Lc0 data, but how do others deal with this?