Chess World to Google Deep Mind..Prove You beat Stockfish 8!

[syzygy]

Me too, Brendan. I am a scientific worker myself, and there's nothing scientific in the way the experiment was conducted. On one side, you have a known entity that everybody can check, on the other, an unknown quantity, a pseudo-scientific paper, an uncalibrated testing behind closed doors, no possibility to repeat the experiment...

A lot of science cannot be easily repeated. If someone reports about research that took 20 years and a couple of thousand volunteers to complete, there is no practical way to "repeat the experiment". What you can do is check the description of the experiment to see if the methods used were sound.

In the case of AlphaZero, the 100-game match between the fully trained AlphaZero and SF8 can be described in just a few words. SF8 was running on 64 cores at about 70 Mnps. AlphaZero using 4 TPUs beat SF8 at 1 minute per move 28-0-72. What more is there to say?

Then, I find it a bit worrying: we are so trained to believe everything that is told to us without checking, without thinking twice, that is has become a paradigm. Enough about it, anyway.

What should we doubt? That AlphaZero won 28-0-72?

Why should we suspect that Deep Mind is lying to us? These are the same guys that did AlphaGo. It would be pretty stupid of them to post fake results, because that would come out eventually.

Maybe they are really stupid, but I find it much more likely that they are not. So for the moment I have no reason to distrust them.

[Uri Blass]

<snipped>

In the case of AlphaZero, the 100-game match between the fully trained AlphaZero and SF8 can be described in just a few words. SF8 was running on 64 cores at about 70 Mnps. AlphaZero using 4 TPUs beat SF8 at 1 minute per move 28-0-72. What more is there to say?

There is a lot more to say
For example a pgn of all the games including not only the 100 games that AlphaZero played against SF8 but all the games that alphazero played earlier against istelf.

[syzygy]

<snipped>

In the case of AlphaZero, the 100-game match between the fully trained AlphaZero and SF8 can be described in just a few words. SF8 was running on 64 cores at about 70 Mnps. AlphaZero using 4 TPUs beat SF8 at 1 minute per move 28-0-72. What more is there to say?

There is a lot more to say
For example a pgn of all the games including not only the 100 games that AlphaZero played against SF8 but all the games that alphazero played earlier against istelf.

That would all be very interesting, but I don't see how that is necessary to make the preprint "scientific". (And a preprint with 44 million games would cost a lot of trees to print out. We didn't get all that information for AlphaGo, but I suppose all the AlphaGo papers are equally unscientific?)

[Milos]

That would all be very interesting, but I don't see how that is necessary to make the preprint "scientific". (And a preprint with 44 million games would cost a lot of trees to print out. We didn't get all that information for AlphaGo, but I suppose all the AlphaGo papers are equally unscientific?)

Why publish only 10 games, why not all 28 wins. Is it really problematic to publish 28 pgn's instead of 10?
Why not publish those very few games in Shogi that they lost?
Because they are covering the scam. It's like Romi chess, most of the wins are very similar if not identical.

Why should we suspect that Deep Mind is lying to us? These are the same guys that did AlphaGo. It would be pretty stupid of them to post fake results, because that would come out eventually.

Why? Because Google is exclusively advertising company very well known for unethical behaviour. And chess in not Go, it's very much different.
How you suppose that it will "come out eventually" when results are virtually non-repeatable? Do you think some insider would come out and testify that they cheated???

[matejst]

Dear Ronald,

It's quite clear you read Schopenhauers' "The Art of Being Right". I did too. In this regard, your first sentence is out of context: this experiment didn't last 20 years, and it was conducted in a very opaque way. I don't know much about neural network, hardware, nor programming, but I am very familiar with scientific methodology.

A genuine experiment of that kind would have been announced, broadcasted (it should be possible), the conditions would have been discussed with the SF developers, and all the games of the match would have been available. This way, a true insight of the strength, the achievements of A0 would be possible. It still can be done, btw.

[syzygy]

It's quite clear you read Schopenhauers' "The Art of Being Right". I did too.

I did not.

In this regard, your first sentence is out of context: this experiment didn't last 20 years, and it was conducted in a very opaque way. I don't know much about neural network, hardware, nor programming, but I am very familiar with scientific methodology.

The example was given to illustrate the point that to be scientific, a paper does not need to put the reader in a position to repeat the full experiment.

A genuine experiment of that kind would have been announced, broadcasted (it should be possible), the conditions would have been discussed with the SF developers, and all the games would have been available.

Broadcasting the match or publishing all games would not have made the experiment any more repeatable for us normal people, AlphaZero would have remained "an unknown quantity", and the testing would still have been as "uncalibrated" as it was now. (But I'm not really sure what is "uncalibrated" about it.)

The conditions under which the match was played are known. AlphaZero beat Stockfish 8 under those conditions.

The conditions they used are not ideal if what you want to know is how AlphaZero would do against SF-dev at 40 moves per 2 hours on 64 cores with 64 GB hash with or without TBs and without or without the Cerebellum book, but that does not invalidate the match they did play. The preprint gives us several data points and it is safe to say that they support the conclusion that the AlphaZero approach works surprisingly well for chess.

The data points do not support the conclusions drawn by some of the people on this forum that basically claim that AlphaZero plays chess from another planet. But that is not a claim made in the preprint.

This way, a true insight of the strength, the achievements of A0 would be possible. It still can be done, btw.

It would certainly be interesting for us computer-chess enthousiasts to learn more about AlphaZero's strength.

What you did not answer is what exactly we should not "believe".

[matejst]

Dear Ronald,

You should then read Schopenhauer. I warmly recommend this small book to your attention.

The example was given to illustrate the point that to be scientific, a paper does not need to put the reader in a position to repeat the full experiment.

That's not what I have written. I don't understand why you insist so much on this point of my first post.

On one side, you have a known entity that everybody can check, on the other, an unknown quantity, a pseudo-scientific paper, an uncalibrated testing behind closed doors, no possibility to repeat the experiment...

That's precisely what I wrote. There's no explicit semantic relation between "a pseudo-scientific paper" and "no possibility to repeat the experiment".

On the other side, "an unknown quantity", "a pseudo-scientific paper", "an uncalibrated testing behind closed doors", "no possibility to repeat the experiment" seems to be true. The reasons why the paper is pseudo-scientific are, without any doubt, clear to the brilliant scientist and mathematician you are yourself.

[syzygy]

The example was given to illustrate the point that to be scientific, a paper does not need to put the reader in a position to repeat the full experiment.

That's not what I have written. I don't understand why you insist so much on this point of my first post.

This is what you wrote:

I am a scientific worker myself, and there's nothing scientific in the way the experiment was conducted. On one side, you have a known entity that everybody can check, on the other, an unknown quantity, a pseudo-scientific paper, an uncalibrated testing behind closed doors, no possibility to repeat the experiment...

You say there's nothing scientific, so according to you the paper was not just "pseudo-scientific" but indeed unscientific. Now I start to look for an argument that you give to back up that statement. The only real thing I can find is "no possibility to repeat the experiment". Indeed I attach no importance to your claim of being a scientific worker yourself, because you will agree with me that it's a bit too easy to say "I am the expert here and believe you me, there's nothing scientific in what Deep Mind did there".

The reasons why the paper is pseudo-scientific are, without any doubt, clear to the brilliant scientist and mathematician you are yourself.

OK, this is a waste of time.

And clearly your have no intention to tell us what it is that we should not "believe".

[matejst]

Of course this is a waste of time.

[Adam Hair]

That would all be very interesting, but I don't see how that is necessary to make the preprint "scientific". (And a preprint with 44 million games would cost a lot of trees to print out. We didn't get all that information for AlphaGo, but I suppose all the AlphaGo papers are equally unscientific?)

Why publish only 10 games, why not all 28 wins. Is it really problematic to publish 28 pgn's instead of 10?
Why not publish those very few games in Shogi that they lost?
Because they are covering the scam. It's like Romi chess, most of the wins are very similar if not identical.

Perhaps because they felt it was a sufficient sample? It could be that simple.

Why should we suspect that Deep Mind is lying to us? These are the same guys that did AlphaGo. It would be pretty stupid of them to post fake results, because that would come out eventually.

Why? Because Google is exclusively advertising company very well known for unethical behaviour. And chess in not Go, it's very much different.
How you suppose that it will "come out eventually" when results are virtually non-repeatable? Do you think some insider would come out and testify that they cheated???

Do you believe they faked AlphaGo? If not, then why so readily believe that the AlphaZero experiment is a sham? After all, it makes more sense to not publicize the experiment if it did not turn out well. Why risk the embarassment?