Alphazero news

[Robert Pope]

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

What makes you believe that?

It is a question of philosophy. As 100% of the practical use of chess engines consists of analysis one can argue that a chess engine should be able to play good chess in any (reasonable) position...

That statement of Matthew is not true. A0 cannot play always as it likes the openings.

Yes, I agree with Kai. Chess is not like that. No one or thing can "always play into closed openings no matter what the opponent does". That claim is not true.

And again, it wasn't a "claim".

"If I could always pick the best move, I would be the best chess player in the world." Clearly a true statement, but I'm not claiming to be the best chess player in the world in that statement.

[jp]

And again, it wasn't a "claim".

It clearly was being claimed by Matthew that A0 can do that. Clearly.
Your statement is also clearly not claiming you can do that.

[Laskos]

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

What makes you believe that?

It is a question of philosophy. As 100% of the practical use of chess engines consists of analysis one can argue that a chess engine should be able to play good chess in any (reasonable) position...

That statement of Matthew is not true. A0 cannot play always as it likes the openings.

Yes, I agree with Kai. Chess is not like that. No one or thing can "always play into closed openings no matter what the opponent does". That claim is not true.

And again, it wasn't a "claim".

"If I could always pick the best move, I would be the best chess player in the world." Clearly a true statement, but I'm not claiming to be the best chess player in the world in that statement.

My English is bad, the context was:

The TCEC openings are all open and tactical openings, favouring SF. Why do you say they are more reliable?

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

It seems to me at least a justification to use favorable to A0 openings, because it's quite possible that A0 steers the openings into favorable ones. Am I missing something? I used a book for SF10, showing that it's far from the truth that A0 can steer every opening the way it likes. So, there is no justification to use favorable to A0 openings. And there is no "supreme" argument to not disturb A0 from playing by its own from Initial Board position, as "it knows better what to do". I let Lc0 play as it wants from the start, and against SF10 + book it performed very poorly compared to the picked openings from the paper. Also, I did have diversity with the book, which they did not have in all their matches aside the TCEC match. I really cannot understand how one can take very seriously a match of 1000 games from 1 position.

Yes, it was maybe not a complete claim, but it was used as an argument to justify the chosen methodology.
Anyway, all this is maybe not that important, A0 is here, Lc0 is here, they are great, the paper is great, so I don't want to be too confrontational.

[jp]

My English is bad, the context was:

The TCEC openings are all open and tactical openings, favouring SF. Why do you say they are more reliable?

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

It seems to me at least a justification to use favorable to A0 openings, because it's quite possible that A0 steers the openings into favorable ones. Am I missing something?

Yes, it was not a complete claim, but it was used as an argument to justify the chosen methodology.

No, you're not missing something. Your English is fine. The context was very, very clear. Even the context of just the sentence on its own was enough ("why should its performance on open openings be reflected in its Elo ratings?").

Robert's example sentence was totally different, and I'm sure he knows that.

[matthewlai]

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

What makes you believe that?

It is a question of philosophy. As 100% of the practical use of chess engines consists of analysis one can argue that a chess engine should be able to play good chess in any (reasonable) position...

That statement of Matthew is not true. A0 cannot play always as it likes the openings.

Yes, I agree with Kai. Chess is not like that. No one or thing can "always play into closed openings no matter what the opponent does". That claim is not true.

And again, it wasn't a "claim".

"If I could always pick the best move, I would be the best chess player in the world." Clearly a true statement, but I'm not claiming to be the best chess player in the world in that statement.

My English is bad, the context was:

The TCEC openings are all open and tactical openings, favouring SF. Why do you say they are more reliable?

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

It seems to me at least a justification to use favorable to A0 openings, because it's quite possible that A0 steers the openings into favorable ones. Am I missing something? I used a book for SF10, showing that it's far from the truth that A0 can steer every opening the way it likes. So, there is no justification to use favorable to A0 openings. And there is no "supreme" argument to not disturb A0 from playing by its own from Initial Board position, as "it knows better what to do". I let Lc0 play as it wants from the start, and against SF10 + book it performed very poorly compared to the picked openings from the paper. Also, I did have diversity with the book, which they did not have in all their matches aside the TCEC match. I really cannot understand how one can take very seriously a match of 1000 games from 1 position.

Yes, it was maybe not a complete claim, but it was used as an argument to justify the chosen methodology.
Anyway, all this is maybe not that important, A0 is here, Lc0 is here, they are great, the paper is great, so I don't want to be too confrontational.

It's a general claim that using forced openings introduces bias depending on opening selection, which I think you agree with. My point is that, if there are openings that AZ or SF will never play itself into, including them will affect the result in a way that does not reflect the strengths of the engines at playing the game of chess (from start position). It's the cost of using any opening suite for diversity.

For example, if SF plays 1. e4 openings very poorly as white, but never plays it, should any 1. e4 openings be used to estimate SF's strength as white?

[Dariusz Orzechowski]

If anyone is interested, I analysed results pictured on fig. S4 in the Supplementary Materials ("Chess matches beginning from the 2016 TCEC world championship start positions"). It looks like each of 50 opening positions was played 100 times (50x with AZ White and 50x with Black). End result was +976 -394 =3630 i.e. +40 Elo for AZ. AZ results with White: +773 -14 =1713, with Black: +203 -380 =1917. In 7/50 openings (marked with asterisk) results were very lopsided (> +100 Elo) in AZ favour. If they were not counted, overall AZ performance drops to +26 Elo. In 12/50 openings AZ performed in range 50-100 Elo. 8/50 openings were won by SF8.

Detailed statistics of all 50 openings in order as in fig. S4:

Code: Select all

     AZ White         AZ Black           Total
    w   l   d        w   l   d         w   l   d
-----------------------------------------------------
   14   0   36       2   1   47       16   1   83  
    9   0   41       3   0   47       12   0   88  
   19   0   31       1   1   48       20   1   79  
    7   3   40       4   0   46       11   3   86  
    9   0   41       8   6   36       17   6   77  
    7   0   43       0   7   43        7   7   86  
   19   0   31       2   7   41       21   7   72  
    2   0   48      22   1   27       24   1   75  
   25   0   25      20   4   26       45   4   51  *
   16   0   34       2   4   44       18   4   78  
   19   0   31      12   1   37       31   1   68  *
    8   0   42       2   2   46       10   2   88  
    6   1   43       9   2   39       15   3   82  
   10   0   40       7   7   36       17   7   76  
   20   0   30       0   2   48       20   2   78  
    6   0   44       2   0   48        8   0   92  
   27   0   23       0  46    4       27  46   27  
    3   0   47       0   3   47        3   3   94  
   35   0   15       0   9   41       35   9   56  
   38   0   12       0  28   22       38  28   34  
   21   0   29       1  29   20       22  29   49  
    6   0   44       0   0   50        6   0   94  
    0   0   50       0   0   50        0   0  100  
    7   0   43       1   4   45        8   4   88  
    8   0   42      12   1   37       20   1   79  
   19   6   25       4   7   39       23  13   64  
    6   0   44       3   7   40        9   7   84  
   21   0   29       0   5   45       21   5   74  
    7   0   43       1   6   43        8   6   86  
   33   0   17       1   4   45       34   4   62  *
    9   0   41       0  22   28        9  22   69  
    8   0   42       0   5   45        8   5   87  
   14   0   36      34   0   16       48   0   52  *
   25   0   25       1  23   26       26  23   51  
    6   0   44       0   1   49        6   1   93  
   35   0   15       1  19   30       36  19   45  
   13   0   37       8  22   20       21  22   57  
    8   1   41       1   0   49        9   1   90  
   13   0   37      11   1   38       24   1   75  
    4   1   45       3  13   34        7  14   79  
   35   0   15       3   2   45       38   2   60  *
   19   1   30       0   0   50       19   1   80  
   37   0   13       4   1   45       41   1   58  *
    5   0   45       0  22   28        5  22   73  
    6   1   43       9   0   41       15   1   84  
    1   0   49       2   7   41        3   7   90  
   44   0    6       0  12   38       44  12   44  *
   15   0   35       3  31   16       18  31   51  
   28   0   22       1   3   46       29   3   68  
   21   0   29       3   2   45       24   2   74  
----------------------------------------------------
  +773 -14 =1713  +203 -380 =1917   +976 -394 =3630

[Laskos]

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

What makes you believe that?

It is a question of philosophy. As 100% of the practical use of chess engines consists of analysis one can argue that a chess engine should be able to play good chess in any (reasonable) position...

That statement of Matthew is not true. A0 cannot play always as it likes the openings.

Yes, I agree with Kai. Chess is not like that. No one or thing can "always play into closed openings no matter what the opponent does". That claim is not true.

And again, it wasn't a "claim".

"If I could always pick the best move, I would be the best chess player in the world." Clearly a true statement, but I'm not claiming to be the best chess player in the world in that statement.

My English is bad, the context was:

The TCEC openings are all open and tactical openings, favouring SF. Why do you say they are more reliable?

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

It seems to me at least a justification to use favorable to A0 openings, because it's quite possible that A0 steers the openings into favorable ones. Am I missing something? I used a book for SF10, showing that it's far from the truth that A0 can steer every opening the way it likes. So, there is no justification to use favorable to A0 openings. And there is no "supreme" argument to not disturb A0 from playing by its own from Initial Board position, as "it knows better what to do". I let Lc0 play as it wants from the start, and against SF10 + book it performed very poorly compared to the picked openings from the paper. Also, I did have diversity with the book, which they did not have in all their matches aside the TCEC match. I really cannot understand how one can take very seriously a match of 1000 games from 1 position.

Yes, it was maybe not a complete claim, but it was used as an argument to justify the chosen methodology.
Anyway, all this is maybe not that important, A0 is here, Lc0 is here, they are great, the paper is great, so I don't want to be too confrontational.

It's a general claim that using forced openings introduces bias depending on opening selection, which I think you agree with. My point is that, if there are openings that AZ or SF will never play itself into, including them will affect the result in a way that does not reflect the strengths of the engines at playing the game of chess (from start position). It's the cost of using any opening suite for diversity.

For example, if SF plays 1. e4 openings very poorly as white, but never plays it, should any 1. e4 openings be used to estimate SF's strength as white?

That's true, but the most striking in my examples was letting Lc0 play whatever it wants from the Initial Board position against SF + Book, so Lc0 was put to play Chess by its definition, but it sometimes doesn't like Chess too much, it likes Chess much more in the paper.

[matthewlai]

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

What makes you believe that?

It is a question of philosophy. As 100% of the practical use of chess engines consists of analysis one can argue that a chess engine should be able to play good chess in any (reasonable) position...

That statement of Matthew is not true. A0 cannot play always as it likes the openings.

Yes, I agree with Kai. Chess is not like that. No one or thing can "always play into closed openings no matter what the opponent does". That claim is not true.

And again, it wasn't a "claim".

"If I could always pick the best move, I would be the best chess player in the world." Clearly a true statement, but I'm not claiming to be the best chess player in the world in that statement.

My English is bad, the context was:

The TCEC openings are all open and tactical openings, favouring SF. Why do you say they are more reliable?

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

It seems to me at least a justification to use favorable to A0 openings, because it's quite possible that A0 steers the openings into favorable ones. Am I missing something? I used a book for SF10, showing that it's far from the truth that A0 can steer every opening the way it likes. So, there is no justification to use favorable to A0 openings. And there is no "supreme" argument to not disturb A0 from playing by its own from Initial Board position, as "it knows better what to do". I let Lc0 play as it wants from the start, and against SF10 + book it performed very poorly compared to the picked openings from the paper. Also, I did have diversity with the book, which they did not have in all their matches aside the TCEC match. I really cannot understand how one can take very seriously a match of 1000 games from 1 position.

Yes, it was maybe not a complete claim, but it was used as an argument to justify the chosen methodology.
Anyway, all this is maybe not that important, A0 is here, Lc0 is here, they are great, the paper is great, so I don't want to be too confrontational.

It's a general claim that using forced openings introduces bias depending on opening selection, which I think you agree with. My point is that, if there are openings that AZ or SF will never play itself into, including them will affect the result in a way that does not reflect the strengths of the engines at playing the game of chess (from start position). It's the cost of using any opening suite for diversity.

For example, if SF plays 1. e4 openings very poorly as white, but never plays it, should any 1. e4 openings be used to estimate SF's strength as white?

That's true, but the most striking in my examples was letting Lc0 play whatever it wants from the Initial Board position against SF + Book, so Lc0 was put to play Chess by its definition, but it sometimes doesn't like Chess too much, it likes Chess much more in the paper.

Is that at a reasonably long time control so that there's no significant difference in quality of moves generated by Leela vs book moves?

[Laskos]

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

What makes you believe that?

It is a question of philosophy. As 100% of the practical use of chess engines consists of analysis one can argue that a chess engine should be able to play good chess in any (reasonable) position...

That statement of Matthew is not true. A0 cannot play always as it likes the openings.

Yes, I agree with Kai. Chess is not like that. No one or thing can "always play into closed openings no matter what the opponent does". That claim is not true.

And again, it wasn't a "claim".

"If I could always pick the best move, I would be the best chess player in the world." Clearly a true statement, but I'm not claiming to be the best chess player in the world in that statement.

My English is bad, the context was:

The TCEC openings are all open and tactical openings, favouring SF. Why do you say they are more reliable?

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

It seems to me at least a justification to use favorable to A0 openings, because it's quite possible that A0 steers the openings into favorable ones. Am I missing something? I used a book for SF10, showing that it's far from the truth that A0 can steer every opening the way it likes. So, there is no justification to use favorable to A0 openings. And there is no "supreme" argument to not disturb A0 from playing by its own from Initial Board position, as "it knows better what to do". I let Lc0 play as it wants from the start, and against SF10 + book it performed very poorly compared to the picked openings from the paper. Also, I did have diversity with the book, which they did not have in all their matches aside the TCEC match. I really cannot understand how one can take very seriously a match of 1000 games from 1 position.

Yes, it was maybe not a complete claim, but it was used as an argument to justify the chosen methodology.
Anyway, all this is maybe not that important, A0 is here, Lc0 is here, they are great, the paper is great, so I don't want to be too confrontational.

It's a general claim that using forced openings introduces bias depending on opening selection, which I think you agree with. My point is that, if there are openings that AZ or SF will never play itself into, including them will affect the result in a way that does not reflect the strengths of the engines at playing the game of chess (from start position). It's the cost of using any opening suite for diversity.

For example, if SF plays 1. e4 openings very poorly as white, but never plays it, should any 1. e4 openings be used to estimate SF's strength as white?

That's true, but the most striking in my examples was letting Lc0 play whatever it wants from the Initial Board position against SF + Book, so Lc0 was put to play Chess by its definition, but it sometimes doesn't like Chess too much, it likes Chess much more in the paper.

Is that at a reasonably long time control so that there's no significant difference in quality of moves generated by Leela vs book moves?

No, but I guess the diversity given by a book for SF doesn't vanish even at infinite time control for Leela. The diversity destabilizes Leela, not the sheer quality of book moves as seen by engines. A0 (Lc0) are Chess game definition specialists, more specialized in that definition than regular engines, which are more general solvers. I anyway showed that for Lc0 months ago here. So, one can train on TCEC openings IF one knows them. If not, then not. All in all, this Zero approach seems not a danger for a general practitioner, or that it will discover something completely new. It seems to do high quality specialized tasks which are already known and performed at lower quality.

[Daniel Shawul]

@Mathew Lai

I have some questions:

Does A0 convolution's have bias, or just convolution+normalization ?

How many games roughly does it take to capture piece values say on the 20b net using self play?
It is not clear to me what kind of filter can be used for counting precisely the number of pieces.