Is this SF NN almost like 20 MB book?

[Alayan]

An interesting test to run would be to measure result spread at different time conditions.

Have a 500 openings book (1000 games match) and run 100 (or more) matches at 10s+0.1, 20s+0.2, 1m+0.6s, 3m+1.8s, 10m+6s. Use a few threads to ensure some SMP variation. Self-play so that the theoretical elo difference should be 0 in each match is probably the soundest way to proceed.

Then analyze the data. Higher draw rate should by itself ensure a lower variance in the results, but I'd expect that even after correcting for the impact of draw rate (an easy trick would be to rerun the ultra-bullet test with a different contempt value that produce a similar draw rate to the rapid test) we'll measure a variance reduction.

From this perspective, while the TCEC superfinal is extremely small sample size (only 100 games) and while the results of individual games could definitely flip if they were run again, the overall results aren't that unreliable.