A lot of science cannot be easily repeated. If someone reports about research that took 20 years and a couple of thousand volunteers to complete, there is no practical way to "repeat the experiment". What you can do is check the description of the experiment to see if the methods used were sound.matejst wrote:Me too, Brendan. I am a scientific worker myself, and there's nothing scientific in the way the experiment was conducted. On one side, you have a known entity that everybody can check, on the other, an unknown quantity, a pseudo-scientific paper, an uncalibrated testing behind closed doors, no possibility to repeat the experiment...
In the case of AlphaZero, the 100-game match between the fully trained AlphaZero and SF8 can be described in just a few words. SF8 was running on 64 cores at about 70 Mnps. AlphaZero using 4 TPUs beat SF8 at 1 minute per move 28-0-72. What more is there to say?
What should we doubt? That AlphaZero won 28-0-72?Then, I find it a bit worrying: we are so trained to believe everything that is told to us without checking, without thinking twice, that is has become a paradigm. Enough about it, anyway.
Why should we suspect that Deep Mind is lying to us? These are the same guys that did AlphaGo. It would be pretty stupid of them to post fake results, because that would come out eventually.
Maybe they are really stupid, but I find it much more likely that they are not. So for the moment I have no reason to distrust them.