Lots of conversation..hand wringing and speculation concerning the “results” of a mysterious “in house unsanctioned” chess match between Google’s Deep Mind Alpha Zero and Stockfish 8.
When the news of the “Match Result” broke I happened to be running my annual engine tourney but with all the buzz and controversy figured it best to read some of the numerous articles/opinions
https://www.wired.com/beyond-the-beyond ... stockfish/
https://medium.com/@josecamachocollados ... 66ae1c84f2
http://www.chessdom.com/interview-with- ... e-houdini/
http://www.captainchess.com/alpha-zero- ... tionalism/
https://en.chessbase.com/post/the-futur ... arns-chess
https://www.chess.com/news/view/alphaze ... ish-author
https://www.quantamagazine.org/new-theo ... -20170921/
https://www.reddit.com/r/chess/comments ... kfish_280/
https://en.chessbase.com/post/kasparov- ... g-in-chess
https://www.theregister.co.uk/2017/12/1 ... ai_unfair/
https://www.youtube.com/watch?v=NxtEDLpJoqQ (Alpha Zero Refutes E4?)
https://content.iospress.com/articles/i ... al/icg0012
who is the master..rating chess
https://rjlipton.wordpress.com/2017/12/ ... from-zero/
Alpha Zero Rated 4500?
https://en.chessbase.com/post/alpha-zer ... and-apples
http://www.espn.com/chess/story/_/id/21 ... hess-world
Anand discusses Alpha Zero effect on chess…
and set about trying to make some sort of sense of events. After a bit of searching found a pgn file with 10 of the 100 games played in the “match”, (all wins by Alpha Zero) with
NO drawn games posted! (that means 90 games of the 100 game “match” are missing. Of the 100 games 28 were supposedly “wins” and 72 were drawn. Time losses/wins or program bug
Issues are unknown. Entered all available information into Fritz and reviewed some of the 10 games (used Deep Fritz 14)
Produced a ratings graph based on estimated chess engine
Strengths (Alpha Zero had no rating prior to the match) used
3000 for Stockfish and 2600 for Alpha Zero…Graph gives a +499
score for Alpha Zero giving a performance rating of 3099…review
of the games show Stockfish winning or better at some point in the games…then having difficulty holding/defending tough positions..Opening choice is also a question..How come so many Queens Indian defences and Ruy Lopez lines? What other openings were played? Any time losses? Where are the other games? These and many other questions need answers…We still remember Kasparov claiming the Deep Blue match was “Rigged” then later the Chess “Program” was dismantled…lets hope
that a repeat of the Deep Blue fiasco is not in the cards this time..
Alpha Zero-Stockfish 8 pgn games http://s000.tinyupload.com/?file_id=012 ... 3751416073
Deep Mind Unsanctioned Testing 12-2017
1 AlphaZero *2600 (+499) +28/-0/=72 64.00% 64.0/100
2 Stockfish 8 3000 (-499) +0/-28/=72 36.00% 36.0/100
(100 Games)
*note: AlphaZero had no actual rating at time of Match…2600 (estimated rating of new program)
Chess World to Google Deep Mind..Prove You beat Stockfish 8!
Moderators: hgm, Rebel, chrisw
-
- Posts: 1242
- Joined: Sat Jul 05, 2014 7:54 am
- Location: Southwest USA
-
- Posts: 2526
- Joined: Mon Feb 08, 2016 12:43 am
- Full name: Brendan J Norman
Re: Chess World to Google Deep Mind..Prove You beat Stockfis
Wow I'm suddenly really tired of hearing about AZ
-
- Posts: 364
- Joined: Mon May 14, 2007 8:20 pm
- Full name: Boban Stanojević
Re: Chess World to Google Deep Mind..Prove You beat Stockfis
Me too, Brendan. I am a scientific worker myself, and there's nothing scientific in the way the experiment was conducted. On one side, you have a known entity that everybody can check, on the other, an unknown quantity, a pseudo-scientific paper, an uncalibrated testing behind closed doors, no possibility to repeat the experiment...
Then, I find it a bit worrying: we are so trained to believe everything that is told to us without checking, without thinking twice, that is has become a paradigm. Enough about it, anyway.
Then, I find it a bit worrying: we are so trained to believe everything that is told to us without checking, without thinking twice, that is has become a paradigm. Enough about it, anyway.
-
- Posts: 2526
- Joined: Mon Feb 08, 2016 12:43 am
- Full name: Brendan J Norman
Re: Chess World to Google Deep Mind..Prove You beat Stockfis
Maybe that's the real experiment.matejst wrote:Then, I find it a bit worrying: we are so trained to believe everything that is told to us without checking, without thinking twice, that is has become a paradigm. Enough about it, anyway.
-
- Posts: 2929
- Joined: Sat Jan 22, 2011 12:42 am
- Location: NL
Re: Chess World to Google Deep Mind..Prove You beat Stockfis
The games were played at fixed time per move, so presumably no time losses.
No opening book was used. AlphaZero knows how to play openings (some will say it has an implicit built-in opening book, which I think misrepresents the issue), Stockfish relies on an opening book.
I would suggest that DeepFritz 14 is insufficient to analyse these games. What you see in terms of evaluation could easily be due to the engine (and Stockfish) missing some evaluation terms that AlphaZero implicitly has, thus mis-evaluating the positions. That cannot be answered easily.
Reviewing the drawn games would be useful, and presumably there is no obstacle to theAlphaZero team making these available on-line.
No opening book was used. AlphaZero knows how to play openings (some will say it has an implicit built-in opening book, which I think misrepresents the issue), Stockfish relies on an opening book.
I would suggest that DeepFritz 14 is insufficient to analyse these games. What you see in terms of evaluation could easily be due to the engine (and Stockfish) missing some evaluation terms that AlphaZero implicitly has, thus mis-evaluating the positions. That cannot be answered easily.
Reviewing the drawn games would be useful, and presumably there is no obstacle to theAlphaZero team making these available on-line.
-
- Posts: 4190
- Joined: Wed Nov 25, 2009 1:47 am
Re: Chess World to Google Deep Mind..Prove You beat Stockfis
Ofc that there is an obstacle. A0 has zero variability in search or evaluation. The only variability there comes on 8 threads waiting in the queue of each TPU for evaluation. Since A0 uses UTC and not alpha-beta this has minuscule impact on variability.Evert wrote:Reviewing the drawn games would be useful, and presumably there is no obstacle to theAlphaZero team making these available on-line.
All other randomness comes from SMP search of SF. Considering TC was fixed time per move, it is very probable that many games are actually similar if not even identical. That would be the only logical explanation why they didn't publish even all the winning games. Many wins basically repeat and there are many more almost identical boring draws.
If they published all the games ppl would see the scam.
-
- Posts: 545
- Joined: Tue Jun 06, 2017 4:49 pm
- Location: Italy
Re: Chess World to Google Deep Mind..Prove You beat Stockfis
I thought an open letter by Stockfish Team was needed, because it's clear to many people that truth has been manipulated. But I understand it's not SF style to spend words. They reply with facts.
It'll be very interesting to see how LeelaCZero will work, with (possibly) thousends users sending their games to the server.
It'll be very interesting to see how LeelaCZero will work, with (possibly) thousends users sending their games to the server.
F.S.I. Chess Teacher
-
- Posts: 4190
- Joined: Wed Nov 25, 2009 1:47 am
Re: Chess World to Google Deep Mind..Prove You beat Stockfis
It will work badly if at all.Rodolfo Leoni wrote:It'll be very interesting to see how LeelaCZero will work, with (possibly) thousends users sending their games to the server.
There are not enough computer chess enthusiasts in the whole world willing to donate time to make training even remotely viable.
Even if you managed to somehow perfectly train LC0 NN, it would still be mediocre engine since its NN is significantly weaker and smaller than NN of A0.
So it is just a lost cause.
-
- Posts: 545
- Joined: Tue Jun 06, 2017 4:49 pm
- Location: Italy
Re: Chess World to Google Deep Mind..Prove You beat Stockfis
I disagree. The lost cause would be to make no attempts, no experiments. As a SF enthusiast, I'll spread this new to everybody in my region. I know for sure there'll be a lot of interest, mostly if their efforts will bring lcZero to participate at WCCC.Milos wrote:It will work badly if at all.Rodolfo Leoni wrote:It'll be very interesting to see how LeelaCZero will work, with (possibly) thousends users sending their games to the server.
There are not enough computer chess enthusiasts in the whole world willing to donate time to make training even remotely viable.
Even if you managed to somehow perfectly train LC0 NN, it would still be mediocre engine since its NN is significantly weaker and smaller than NN of A0.
So it is just a lost cause.
And if it'll be a failure, it's a good experience anyway. One can learn a lot from failures. So I'll try to support this initiative in any way.
F.S.I. Chess Teacher
-
- Posts: 1242
- Joined: Sat Jul 05, 2014 7:54 am
- Location: Southwest USA
Re: Chess World to Google Deep Mind..Prove You beat Stockfis
Yes Milos my friend...agreed 100% with your post and your prior positions on this important issue....Thx ARMilos wrote:Ofc that there is an obstacle. A0 has zero variability in search or evaluation. The only variability there comes on 8 threads waiting in the queue of each TPU for evaluation. Since A0 uses UTC and not alpha-beta this has minuscule impact on variability.Evert wrote:Reviewing the drawn games would be useful, and presumably there is no obstacle to theAlphaZero team making these available on-line.
All other randomness comes from SMP search of SF. Considering TC was fixed time per move, it is very probable that many games are actually similar if not even identical. That would be the only logical explanation why they didn't publish even all the winning games. Many wins basically repeat and there are many more almost identical boring draws.
If they published all the games ppl would see the scam.