Cerebellum analysis of the AlphaZero - Stockfish Games

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
clumma
Posts: 177
Joined: Fri Oct 10, 2014 8:05 pm
Location: Berkeley, CA

Re: Cerebellum analysis of the AlphaZero - Stockfish Games

Post by clumma » Tue Dec 19, 2017 12:18 am

Thank you, Zipproth. I was wondering what Cerebellum would say about these games.

I would also like to say, I am ready to buy Cerebellum, even if all features are not available yet. And please PM me if you need a beta tester!

-Carl

User avatar
Rebel
Posts: 4518
Joined: Thu Aug 18, 2011 10:04 am

Re: Cerebellum analysis of the AlphaZero - Stockfish Games

Post by Rebel » Tue Dec 19, 2017 9:15 am

Thomas Zipproth wrote: I calculated the AlphaZero games and added them to Cerebellum, which took some time because Stockfish naturally needed very long calculation times and many added nodes to finally get the (most likely) correct results.

http://www.zipproth.de/Brainfish/Cerebe ... aZero.html

I thought about a way to present a game calculation in a most comprehensive but simple way, and finally ended up with this images.
Of course all side lines are missing, only the best move is additionally displayed, if it is different from the move which was played in the game.

Some conclusions or assumptions gained from the results:

Stockfish's evaluation ist correct most times, but it does not distinguish between a position which is dead draw and other positions which are theoretically also draw, but very complicated with many occassions to make a wrong move. In such positions stockfish sometimes makes errors leading to a loss, even if most times the evaluation of close to zero is correct.
That has amazed me too. AZ never follows the draw SF suggests when there are more than one move that lead to a forced draw, 3-fold repetitions mainly. It seems to know exactly which move to avoid. What other conclusion is there then that position is learned?

Lyudmil Tsvetkov
Posts: 6037
Joined: Tue Jun 12, 2012 10:41 am

Re: Cerebellum analysis of the AlphaZero - Stockfish Games

Post by Lyudmil Tsvetkov » Tue Dec 19, 2017 12:46 pm

Ras wrote:
Lyudmil Tsvetkov wrote:But they are checking facts, while you are just speculating.
Btw., these facts somewhat clash with your claim that 80% of the games were lost right in the opening.

Not qualifying: games 1, 2, 3, 7, 8, 10.
Qualifying: game 4.
Debatable: games 5, 6, 9 (move 18/19 is not really opening anymore, IMO).

Or, that is the alternative conclusion, the opening book wouldn't have changed much because it doesn't see the early losses that you can see.
I don't understand what you are talking about, what facts?
Who claims that at least 4 games have not been lost until move 10 and 4 more close to losing?
If it is Cerebellum, than this is just SF, precisely the engine you should not be using for verification.

Ras
Posts: 1137
Joined: Tue Aug 30, 2016 6:19 pm
Contact:

Re: Cerebellum analysis of the AlphaZero - Stockfish Games

Post by Ras » Tue Dec 19, 2017 1:27 pm

Lyudmil Tsvetkov wrote:I don't understand what you are talking about, what facts?
Take a look at the graphs from the original posting to see at which point they turned.
Who claims that at least 4 games have not been lost until move 10 and 4 more close to losing?
Well of course you sometimes claim that a certain game is lost after move 2 of an established opening - but if it's "Tsvetkov loss" where no opposing engine is actually able to win that by force, it's not really relevant in an engine match. Or, if the issue is that Alpha0 now IS able to win that, then it is the better engine.
If it is Cerebellum, than this is just SF, precisely the engine you should not be using for verification.
Disagree. First, Alpha0 only had self-play, too, so it would be a an obvious measure to check what Stockfish can do on it's own if given enough time. The data indicate that the majority of the losses of these 10 games would have happened anyway. Actually, only game 4 is such an early loss that a book could have prevented that.

Second, Cerebellum was exactly the book a lot of people would have wanted to include because it's the best that Stockfish can have.

Third, what other book would you have suggested if not Cerebellum?

Lyudmil Tsvetkov
Posts: 6037
Joined: Tue Jun 12, 2012 10:41 am

Re: Cerebellum analysis of the AlphaZero - Stockfish Games

Post by Lyudmil Tsvetkov » Tue Dec 19, 2017 7:50 pm

Ras wrote:
Lyudmil Tsvetkov wrote:I don't understand what you are talking about, what facts?
Take a look at the graphs from the original posting to see at which point they turned.
Who claims that at least 4 games have not been lost until move 10 and 4 more close to losing?
Well of course you sometimes claim that a certain game is lost after move 2 of an established opening - but if it's "Tsvetkov loss" where no opposing engine is actually able to win that by force, it's not really relevant in an engine match. Or, if the issue is that Alpha0 now IS able to win that, then it is the better engine.
If it is Cerebellum, than this is just SF, precisely the engine you should not be using for verification.
Disagree. First, Alpha0 only had self-play, too, so it would be a an obvious measure to check what Stockfish can do on it's own if given enough time. The data indicate that the majority of the losses of these 10 games would have happened anyway. Actually, only game 4 is such an early loss that a book could have prevented that.

Second, Cerebellum was exactly the book a lot of people would have wanted to include because it's the best that Stockfish can have.

Third, what other book would you have suggested if not Cerebellum?
Some people spot the loss 1 move before mate... :(
Others know the loss is there on move 2.

For example, 1.g4 loses, but how many on this forum would subscribe to this assertion?
I guess I will be the only one.

Cerebellum still plays 1. e4, and, on 1. e4, 1...e6.
Not much to discuss here.
That is what sent SF most convincingly down the drain: choosing the French, where it lost 39(!) games with black out of 50.
Also never fianchettoeing its bishop kingside, either with white or black, on that only it lost most of the QIDs.

Leo
Posts: 790
Joined: Fri Sep 16, 2016 4:55 pm
Location: USA/Minnesota
Full name: Leo

Re: Cerebellum analysis of the AlphaZero - Stockfish Games

Post by Leo » Tue Dec 19, 2017 9:41 pm

clumma wrote:Thank you, Zipproth. I was wondering what Cerebellum would say about these games.

I would also like to say, I am ready to buy Cerebellum, even if all features are not available yet. And please PM me if you need a beta tester!

-Carl
Many people want to buy it.
BrainFish-2 190310 bmi2 Elo 3576.

User avatar
mhull
Posts: 12263
Joined: Wed Mar 08, 2006 8:02 pm
Location: Dallas, Texas
Full name: Matthew Hull

Re: Cerebellum analysis of the AlphaZero - Stockfish Games

Post by mhull » Wed Dec 20, 2017 12:05 am

Rebel wrote:
Thomas Zipproth wrote: I calculated the AlphaZero games and added them to Cerebellum, which took some time because Stockfish naturally needed very long calculation times and many added nodes to finally get the (most likely) correct results.

http://www.zipproth.de/Brainfish/Cerebe ... aZero.html

I thought about a way to present a game calculation in a most comprehensive but simple way, and finally ended up with this images.
Of course all side lines are missing, only the best move is additionally displayed, if it is different from the move which was played in the game.

Some conclusions or assumptions gained from the results:

Stockfish's evaluation ist correct most times, but it does not distinguish between a position which is dead draw and other positions which are theoretically also draw, but very complicated with many occassions to make a wrong move. In such positions stockfish sometimes makes errors leading to a loss, even if most times the evaluation of close to zero is correct.
That has amazed me too. AZ never follows the draw SF suggests when there are more than one move that lead to a forced draw, 3-fold repetitions mainly. It seems to know exactly which move to avoid. What other conclusion is there then that position is learned?
This question seems a false dichotomy to me. The assumption is that a new program cannot see beyond what the current standard can know. But I have always had high expectations of NNs, that they could see be trained to find patterns that human programmers might entirely miss. And since algos are humanly derived...
NNs learn what we cannot see. That is the promise of NNs. That's why I'm less skeptical about this.
Matthew Hull

Ras
Posts: 1137
Joined: Tue Aug 30, 2016 6:19 pm
Contact:

Re: Cerebellum analysis of the AlphaZero - Stockfish Games

Post by Ras » Wed Dec 20, 2017 1:07 am

Rebel wrote:AZ never follows the draw SF suggests when there are more than one move that lead to a forced draw, 3-fold repetitions mainly. It seems to know exactly which move to avoid. What other conclusion is there then that position is learned?
That AZ sees something better than a draw. That was easy.

User avatar
Rebel
Posts: 4518
Joined: Thu Aug 18, 2011 10:04 am

Re: Cerebellum analysis of the AlphaZero - Stockfish Games

Post by Rebel » Wed Dec 20, 2017 2:59 pm

Ras wrote:
Rebel wrote:AZ never follows the draw SF suggests when there are more than one move that lead to a forced draw, 3-fold repetitions mainly. It seems to know exactly which move to avoid. What other conclusion is there then that position is learned?
That AZ sees something better than a draw. That was easy.
Yep, 2 (or more) moves that lead to a draw, the SF move played has a W/L/D value of 50% hence they play the other one with +50.1% established during the training. Easy too as well.

CheckersGuy
Posts: 273
Joined: Wed Aug 24, 2016 7:49 pm

Re: Cerebellum analysis of the AlphaZero - Stockfish Games

Post by CheckersGuy » Wed Dec 20, 2017 4:07 pm

A neural network is not an opening book. Nough said :lol:

Post Reply