Chess World to Google Deep Mind..Prove You beat Stockfish 8!

[supersharp77]

Lots of conversation..hand wringing and speculation concerning the “results” of a mysterious “in house unsanctioned” chess match between Google’s Deep Mind Alpha Zero and Stockfish 8.
When the news of the “Match Result” broke I happened to be running my annual engine tourney but with all the buzz and controversy figured it best to read some of the numerous articles/opinions

https://www.wired.com/beyond-the-beyond ... stockfish/

https://medium.com/@josecamachocollados ... 66ae1c84f2


http://www.chessdom.com/interview-with- ... e-houdini/

http://www.captainchess.com/alpha-zero- ... tionalism/

https://en.chessbase.com/post/the-futur ... arns-chess


https://www.chess.com/news/view/alphaze ... ish-author

https://www.quantamagazine.org/new-theo ... -20170921/

https://www.reddit.com/r/chess/comments ... kfish_280/


https://en.chessbase.com/post/kasparov- ... g-in-chess

https://www.theregister.co.uk/2017/12/1 ... ai_unfair/


https://www.youtube.com/watch?v=NxtEDLpJoqQ (Alpha Zero Refutes E4?)


https://content.iospress.com/articles/i ... al/icg0012
who is the master..rating chess

https://rjlipton.wordpress.com/2017/12/ ... from-zero/
Alpha Zero Rated 4500?

https://en.chessbase.com/post/alpha-zer ... and-apples


http://www.espn.com/chess/story/_/id/21 ... hess-world

Anand discusses Alpha Zero effect on chess…



and set about trying to make some sort of sense of events. After a bit of searching found a pgn file with 10 of the 100 games played in the “match”, (all wins by Alpha Zero) with
NO drawn games posted! (that means 90 games of the 100 game “match” are missing. Of the 100 games 28 were supposedly “wins” and 72 were drawn. Time losses/wins or program bug
Issues are unknown. Entered all available information into Fritz and reviewed some of the 10 games (used Deep Fritz 14)
Produced a ratings graph based on estimated chess engine
Strengths (Alpha Zero had no rating prior to the match) used
3000 for Stockfish and 2600 for Alpha Zero…Graph gives a +499
score for Alpha Zero giving a performance rating of 3099…review
of the games show Stockfish winning or better at some point in the games…then having difficulty holding/defending tough positions..Opening choice is also a question..How come so many Queens Indian defences and Ruy Lopez lines? What other openings were played? Any time losses? Where are the other games? These and many other questions need answers…We still remember Kasparov claiming the Deep Blue match was “Rigged” then later the Chess “Program” was dismantled…lets hope
that a repeat of the Deep Blue fiasco is not in the cards this time..


Alpha Zero-Stockfish 8 pgn games http://s000.tinyupload.com/?file_id=012 ... 3751416073

Deep Mind Unsanctioned Testing 12-2017


1 AlphaZero *2600 (+499) +28/-0/=72 64.00% 64.0/100
2 Stockfish 8 3000 (-499) +0/-28/=72 36.00% 36.0/100

(100 Games)




*note: AlphaZero had no actual rating at time of Match…2600 (estimated rating of new program)

[BrendanJNorman]

Wow I'm suddenly really tired of hearing about AZ

[matejst]

Me too, Brendan. I am a scientific worker myself, and there's nothing scientific in the way the experiment was conducted. On one side, you have a known entity that everybody can check, on the other, an unknown quantity, a pseudo-scientific paper, an uncalibrated testing behind closed doors, no possibility to repeat the experiment...

Then, I find it a bit worrying: we are so trained to believe everything that is told to us without checking, without thinking twice, that is has become a paradigm. Enough about it, anyway.

[BrendanJNorman]

Then, I find it a bit worrying: we are so trained to believe everything that is told to us without checking, without thinking twice, that is has become a paradigm. Enough about it, anyway.

Maybe that's the real experiment.

[Evert]

The games were played at fixed time per move, so presumably no time losses.
No opening book was used. AlphaZero knows how to play openings (some will say it has an implicit built-in opening book, which I think misrepresents the issue), Stockfish relies on an opening book.
I would suggest that DeepFritz 14 is insufficient to analyse these games. What you see in terms of evaluation could easily be due to the engine (and Stockfish) missing some evaluation terms that AlphaZero implicitly has, thus mis-evaluating the positions. That cannot be answered easily.
Reviewing the drawn games would be useful, and presumably there is no obstacle to theAlphaZero team making these available on-line.

[Milos]

Reviewing the drawn games would be useful, and presumably there is no obstacle to theAlphaZero team making these available on-line.

Ofc that there is an obstacle. A0 has zero variability in search or evaluation. The only variability there comes on 8 threads waiting in the queue of each TPU for evaluation. Since A0 uses UTC and not alpha-beta this has minuscule impact on variability.
All other randomness comes from SMP search of SF. Considering TC was fixed time per move, it is very probable that many games are actually similar if not even identical. That would be the only logical explanation why they didn't publish even all the winning games. Many wins basically repeat and there are many more almost identical boring draws.
If they published all the games ppl would see the scam.

[Rodolfo Leoni]

I thought an open letter by Stockfish Team was needed, because it's clear to many people that truth has been manipulated. But I understand it's not SF style to spend words. They reply with facts.

It'll be very interesting to see how LeelaCZero will work, with (possibly) thousends users sending their games to the server.

[Milos]

It'll be very interesting to see how LeelaCZero will work, with (possibly) thousends users sending their games to the server.

It will work badly if at all.
There are not enough computer chess enthusiasts in the whole world willing to donate time to make training even remotely viable.
Even if you managed to somehow perfectly train LC0 NN, it would still be mediocre engine since its NN is significantly weaker and smaller than NN of A0.
So it is just a lost cause.

[Rodolfo Leoni]

It'll be very interesting to see how LeelaCZero will work, with (possibly) thousends users sending their games to the server.

It will work badly if at all.
There are not enough computer chess enthusiasts in the whole world willing to donate time to make training even remotely viable.
Even if you managed to somehow perfectly train LC0 NN, it would still be mediocre engine since its NN is significantly weaker and smaller than NN of A0.
So it is just a lost cause.

I disagree. The lost cause would be to make no attempts, no experiments. As a SF enthusiast, I'll spread this new to everybody in my region. I know for sure there'll be a lot of interest, mostly if their efforts will bring lcZero to participate at WCCC.

And if it'll be a failure, it's a good experience anyway. One can learn a lot from failures. So I'll try to support this initiative in any way.

[supersharp77]

Reviewing the drawn games would be useful, and presumably there is no obstacle to theAlphaZero team making these available on-line.

Ofc that there is an obstacle. A0 has zero variability in search or evaluation. The only variability there comes on 8 threads waiting in the queue of each TPU for evaluation. Since A0 uses UTC and not alpha-beta this has minuscule impact on variability.
All other randomness comes from SMP search of SF. Considering TC was fixed time per move, it is very probable that many games are actually similar if not even identical. That would be the only logical explanation why they didn't publish even all the winning games. Many wins basically repeat and there are many more almost identical boring draws.
If they published all the games ppl would see the scam.

Yes Milos my friend...agreed 100% with your post and your prior positions on this important issue....Thx AR