Cerebellum analysis of the AlphaZero - Stockfish Games

[clumma]

Thank you, Zipproth. I was wondering what Cerebellum would say about these games.

I would also like to say, I am ready to buy Cerebellum, even if all features are not available yet. And please PM me if you need a beta tester!

-Carl

[Rebel]

I calculated the AlphaZero games and added them to Cerebellum, which took some time because Stockfish naturally needed very long calculation times and many added nodes to finally get the (most likely) correct results.

http://www.zipproth.de/Brainfish/Cerebe ... aZero.html

I thought about a way to present a game calculation in a most comprehensive but simple way, and finally ended up with this images.
Of course all side lines are missing, only the best move is additionally displayed, if it is different from the move which was played in the game.

Some conclusions or assumptions gained from the results:

Stockfish's evaluation ist correct most times, but it does not distinguish between a position which is dead draw and other positions which are theoretically also draw, but very complicated with many occassions to make a wrong move. In such positions stockfish sometimes makes errors leading to a loss, even if most times the evaluation of close to zero is correct.

That has amazed me too. AZ never follows the draw SF suggests when there are more than one move that lead to a forced draw, 3-fold repetitions mainly. It seems to know exactly which move to avoid. What other conclusion is there then that position is learned?

[Lyudmil Tsvetkov]

But they are checking facts, while you are just speculating.

Btw., these facts somewhat clash with your claim that 80% of the games were lost right in the opening.

Not qualifying: games 1, 2, 3, 7, 8, 10.
Qualifying: game 4.
Debatable: games 5, 6, 9 (move 18/19 is not really opening anymore, IMO).

Or, that is the alternative conclusion, the opening book wouldn't have changed much because it doesn't see the early losses that you can see.

I don't understand what you are talking about, what facts?
Who claims that at least 4 games have not been lost until move 10 and 4 more close to losing?
If it is Cerebellum, than this is just SF, precisely the engine you should not be using for verification.

[Ras]

I don't understand what you are talking about, what facts?

Take a look at the graphs from the original posting to see at which point they turned.

Who claims that at least 4 games have not been lost until move 10 and 4 more close to losing?

Well of course you sometimes claim that a certain game is lost after move 2 of an established opening - but if it's "Tsvetkov loss" where no opposing engine is actually able to win that by force, it's not really relevant in an engine match. Or, if the issue is that Alpha0 now IS able to win that, then it is the better engine.

If it is Cerebellum, than this is just SF, precisely the engine you should not be using for verification.

Disagree. First, Alpha0 only had self-play, too, so it would be a an obvious measure to check what Stockfish can do on it's own if given enough time. The data indicate that the majority of the losses of these 10 games would have happened anyway. Actually, only game 4 is such an early loss that a book could have prevented that.

Second, Cerebellum was exactly the book a lot of people would have wanted to include because it's the best that Stockfish can have.

Third, what other book would you have suggested if not Cerebellum?

[Lyudmil Tsvetkov]

I don't understand what you are talking about, what facts?

Take a look at the graphs from the original posting to see at which point they turned.

Who claims that at least 4 games have not been lost until move 10 and 4 more close to losing?

Well of course you sometimes claim that a certain game is lost after move 2 of an established opening - but if it's "Tsvetkov loss" where no opposing engine is actually able to win that by force, it's not really relevant in an engine match. Or, if the issue is that Alpha0 now IS able to win that, then it is the better engine.

If it is Cerebellum, than this is just SF, precisely the engine you should not be using for verification.

Disagree. First, Alpha0 only had self-play, too, so it would be a an obvious measure to check what Stockfish can do on it's own if given enough time. The data indicate that the majority of the losses of these 10 games would have happened anyway. Actually, only game 4 is such an early loss that a book could have prevented that.

Second, Cerebellum was exactly the book a lot of people would have wanted to include because it's the best that Stockfish can have.

Third, what other book would you have suggested if not Cerebellum?

Some people spot the loss 1 move before mate...
Others know the loss is there on move 2.

For example, 1.g4 loses, but how many on this forum would subscribe to this assertion?
I guess I will be the only one.

Cerebellum still plays 1. e4, and, on 1. e4, 1...e6.
Not much to discuss here.
That is what sent SF most convincingly down the drain: choosing the French, where it lost 39(!) games with black out of 50.
Also never fianchettoeing its bishop kingside, either with white or black, on that only it lost most of the QIDs.

[Leo]

Thank you, Zipproth. I was wondering what Cerebellum would say about these games.

I would also like to say, I am ready to buy Cerebellum, even if all features are not available yet. And please PM me if you need a beta tester!

-Carl

Many people want to buy it.

[mhull]

I calculated the AlphaZero games and added them to Cerebellum, which took some time because Stockfish naturally needed very long calculation times and many added nodes to finally get the (most likely) correct results.

http://www.zipproth.de/Brainfish/Cerebe ... aZero.html

I thought about a way to present a game calculation in a most comprehensive but simple way, and finally ended up with this images.
Of course all side lines are missing, only the best move is additionally displayed, if it is different from the move which was played in the game.

Some conclusions or assumptions gained from the results:

Stockfish's evaluation ist correct most times, but it does not distinguish between a position which is dead draw and other positions which are theoretically also draw, but very complicated with many occassions to make a wrong move. In such positions stockfish sometimes makes errors leading to a loss, even if most times the evaluation of close to zero is correct.

That has amazed me too. AZ never follows the draw SF suggests when there are more than one move that lead to a forced draw, 3-fold repetitions mainly. It seems to know exactly which move to avoid. What other conclusion is there then that position is learned?

This question seems a false dichotomy to me. The assumption is that a new program cannot see beyond what the current standard can know. But I have always had high expectations of NNs, that they could see be trained to find patterns that human programmers might entirely miss. And since algos are humanly derived...
NNs learn what we cannot see. That is the promise of NNs. That's why I'm less skeptical about this.

[Ras]

AZ never follows the draw SF suggests when there are more than one move that lead to a forced draw, 3-fold repetitions mainly. It seems to know exactly which move to avoid. What other conclusion is there then that position is learned?

That AZ sees something better than a draw. That was easy.

[Rebel]

AZ never follows the draw SF suggests when there are more than one move that lead to a forced draw, 3-fold repetitions mainly. It seems to know exactly which move to avoid. What other conclusion is there then that position is learned?

That AZ sees something better than a draw. That was easy.

Yep, 2 (or more) moves that lead to a draw, the SF move played has a W/L/D value of 50% hence they play the other one with +50.1% established during the training. Easy too as well.

[CheckersGuy]

A neural network is not an opening book. Nough said