The thing that I gleaned from the paper is that the new approach looks much better than the old one.Gian-Carlo Pascutto wrote:While there are some things in the paper that can be criticized, your criticism itself is just as flawed.This is total crap, of corse, becasue you cannot increase by training with less games it takes to measure the ELO gain itself, and in 100 games you measure nothing, so this is just another piece of crap.
Just to be more clear, if the engine increased by 500 ELO points in 100 games it is _theoretically_ only for pure luck.
ELO gains cannot be measured exactly without an infinite number of games, so "measuring ELO gain" can realistically only be done with some error margin.
Measuring an improvement of 500 ELO in 100 games is going to be statistically significant. The error margin might be 300 ELO, but this means worst case there is still an improvement of 200 ELO. An ELO improvement of 500 ELO over 100 games means a score of something like 5 - 95!
So, if we don't believe it, we can try it ourselves.