Self-distillation NNUE: the student/teacher transfer ratio drops every generation and plateaus after three

[Spunc595]

Hi everyone,

I've been bootstrapping an NNUE for my engine with a strict self-imposed rule: every position and label must come from my engine's own search. No external evaluator anywhere in the pipeline. I've now hit a wall I can measure but can't really explain, and I'd rather ask here than keep losing nights over it.

Setup per generation: self-play at 3k nodes, ~3M unique positions, labels (eval + best move) from my search at 20k nodes using the previous generation's net as teacher. Net architecture is 768x4 king buckets -> 1024, SCReLU, akimbo-style. Training saturates very quickly and early-stops.

The ladder flattens fast:

Code: Select all

           rho vs Stockfish      step
        (static, same 2,000 positions)

  gen1        0.587
  gen2        0.679             +0.092
  gen3        0.700             +0.021

It looks like the student is capturing less of its teacher with each step. Teacher measured in search at 20k nodes, student measured static:

Code: Select all

  gen2:   0.679 / 0.825  =  0.823
  gen3:   0.700 / 0.876  =  0.800

So the teacher gains +0.051, but the student only gets +0.021. My guess is that as the teacher gets stronger, more of its strength comes from search/tree expansion, which a static net just can't represent — making a growing part of the label look like pure noise to the student. But that's just a guess.

Two things I already tested and ruled out:

Code: Select all

  Capacity:     1024 -> 2048 hidden = 0.695 (no gain, actually slightly
                worse) and 26% lower nps. Same dataset, only L1 changed.

  Data volume:  val loss bottoms out at epoch 3 with 3.1M positions,
                and was epoch 4 with 2.97M. More data just meant faster
                saturation.

And the cost in Elo is heavy (measured on the same binary, only changing the net file, 500 games per pairing at 20+0.2):

gen3 is -759 ± 58 against the akimbo net, and -277 ± 36 against my own old net that was trained on external labels. gen3 did not win a single one of the 500 games against akimbo.

A few questions for the forum:

1. Is this declining student/teacher transfer ratio something you've seen before? Is it just how self-distillation behaves, or a sign that I'm doing something wrong?

2. What scale does this usually need (positions per iteration, total iterations)? I'm starting to think 3M per generation is off by an order of magnitude or two.

3. Quiet-position filtering: I currently only discard positions in check. Around 19.7% of my positions have a capture as best move. Is filtering those out standard practice, and does it actually make a big difference?

4. Is it better to label the static eval of the PV leaf instead of the root search score? That would address the "unlearnable search noise" problem, but maybe I'm just labeling the wrong node.

Everything is open source, including the raw measurements and a couple of predictions I registered in advance and got completely wrong: https://github.com/Spunc595/Luna-CE-NNUE

Thanks,

Daniele