Stats, Testing, Quit Early and Dumb Luck

[jwes]

Of course, the real issue is not TC but average depth of search. Thus, different programs are affected differently. Still, the question is at what
speed can we trust statistics or must we all purchase clusters?

I think that any effect would be the other way, i.e. it could be that at longer time controls the error bars are smaller than statistics would predict. Statistical theory takes into account all possible "luck". If the error bars are wider than predicted, it would necessarily mean there is some correlation between different runs.