Alphazero news

[yanquis1972]

No, that sounds sensible to me. Without knowing the strength tho (how many elo does an unassisted SF lose to it? What’s the typical SF10 eval out of book compared to the start position vs Lc0 and In self play?) I’d still advocate for truncating after a few moves.

I don’t know that the deepmind set were cherrypicked, but I think some (reti and QPG come to mind) could be replaced with positive effect. I might even try simplifying it further (1. e4/d4/c4, then force some lines that don’t appear.)

[Milos]

Cerebelleum best book lines give even worse diversification to SF8 than SF8 itself on 44 threads. I have no confidence in that result. "Human Openings" given in A0 preprint (those 12) were heavily favoring A0, and I have again no confidence in that result. I don't even know how many openings they choose, the same 12?

Yes, the same 12. The openings the most commonly played openings from a large set of high level human games (I don't remember which set off the top of my head). There is no cherry-picking. They were the top 12. If they favour AZ, that just means AZ plays common human openings better.

Since no one actually comments on this, I will.
No, these 12 are not the most commonly played openings from a large set of high level human games and how conveniently you don't remember from which set.
Here is a simple proof, take Reti from the preprint (1.Nf3 Nf3 2.c4 e6 3. d4 d5 4. Nc3 Be7 5. Bf4 O-O). The actual opening used is 5 moves deep. After 2.c4 most further moves by both black and white are like 2nd and 3rd possibility if you look at any large database of high level human openings. So from 70k games in database after 2.c4 we end up at only 6k after 5. .. O-O.
To me these seems like deliberately chosen 12 positions from 12 most popular openings that have nothing to do with popularity between humans or representably of chess. They were only selected to give A0 advantage it needed to win.

[yanquis1972]

Apologies if this was accounted for in kais test but a factor I didn’t consider is that an opening book is a form of time odds against a bookless opponent. Eg in an ongoing game SF10 used about 90 seconds by move 6 or. 7 in a 5+5 game before SF10book had to touch the clock.

[jp]

Cerebelleum best book lines give even worse diversification to SF8 than SF8 itself on 44 threads. I have no confidence in that result. "Human Openings" given in A0 preprint (those 12) were heavily favoring A0, and I have again no confidence in that result. I don't even know how many openings they choose, the same 12?

Yes, the same 12. The openings the most commonly played openings from a large set of high level human games (I don't remember which set off the top of my head). There is no cherry-picking. They were the top 12. If they favour AZ, that just means AZ plays common human openings better.

Since no one actually comments on this, I will.
No, these 12 are not the most commonly played openings from a large set of high level human games and how conveniently you don't remember from which set.
Here is a simple proof, take Reti from the preprint

They are definitely not high level human games. They are not even medium level. They are low level human games (meaning games of any quality including very low and high, but of course that means far more bad games than good games). I checked the free online database they used. e.g. it shows > 75% of the games in the database for the Reti 1.Nf3 Nf6 are by very weak players -- must be amateurs.

[Uri Blass]

Cerebelleum best book lines give even worse diversification to SF8 than SF8 itself on 44 threads. I have no confidence in that result. "Human Openings" given in A0 preprint (those 12) were heavily favoring A0, and I have again no confidence in that result. I don't even know how many openings they choose, the same 12?

Yes, the same 12. The openings the most commonly played openings from a large set of high level human games (I don't remember which set off the top of my head). There is no cherry-picking. They were the top 12. If they favour AZ, that just means AZ plays common human openings better.

Since no one actually comments on this, I will.
No, these 12 are not the most commonly played openings from a large set of high level human games and how conveniently you don't remember from which set.
Here is a simple proof, take Reti from the preprint (1.Nf3 Nf3 2.c4 e6 3. d4 d5 4. Nc3 Be7 5. Bf4 O-O). The actual opening used is 5 moves deep. After 2.c4 most further moves by both black and white are like 2nd and 3rd possibility if you look at any large database of high level human openings. So from 70k games in database after 2.c4 we end up at only 6k after 5. .. O-O.
To me these seems like deliberately chosen 12 positions from 12 most popular openings that have nothing to do with popularity between humans or representably of chess. They were only selected to give A0 advantage it needed to win.

1)I do not like this subject to have personal attacks.
It is enough to prove your point that the openings are not based on high level human games.

I think that it is better not to write things like
"how conveniently you don't remember from which set."

People can make mistakes and I prefer to assume that they do not make them intentionally

2)I would like to know who is responsible for the graph at page 7 that showed that alphazero scales better than stockfish.

https://arxiv.org/pdf/1712.01815.pdf


I read later that stockfish was practically constant in the test but the graph does not show constant elo for stockfish and show that stockfish also get higher elo with more time.

[Laskos]

And 3) Book saves time.

That is true. This can be compensated for by giving Lc0 more time so that when SF comes out of book, they have roughly equal time left.

the main effect againt Lc0 is not the quality of the book moves, but their diversity which allows to put Lc0 in less familiar to it positions

How do you know this?

Many games exited with close to 0.00 evals out of the book, evals shown by both engines, but often Lc0 mishandled them to lose, much more often than viceversa.

Old results ar 1min + 1s TC (Lc0 on RTX 2070, SF10 on 4 i7 threads):

Lc0 No Book vs SF10 BookX.bin:
Score of lc0_v191_11261 vs SF10: 5 - 16 - 19 [0.362] 40
Elo difference: -98.07 +/- 79.65


Lc0 No Book vs SF10 No Book (Initial Board position):
Score of lc0_v19_11261 vs SF10: 12 - 6 - 22 [0.575] 40
Elo difference: 52.51 +/- 73.05

==================================


New result at 4min + 4s TC:

Lc0 No Book vs SF10 BookX.bin:
Score of lc0_v19_11261 vs SF10: 2 - 14 - 24 [0.350] 40
Elo difference: -107.54 +/- 66.82

Even worse than before, but the number of games is small (40).

Lc0 No Book vs SF10 No Book (Initial Board position):
Score of lc0_v19_11261 vs SF10: 8 - 5 - 27 (0.537) 40
Elo difference: 26.11 +/- 61.76

So, at 4x time control, the difference is pretty stable.
I think the burden is on you to show that in your conditions, diversified BookX.bin book or some other even small good diversified books don't make much difference. In fact, this is the last argument left, that at long TC and big hardware, this difference vanishes, a thing which I highly doubt. Anyway, playing 1000 games from 1 Initial Board position at very long time control seems a bit hilarious to me. Cannot you try some 200 with a diversified book for SF?

In this interesting experiment I think that the confrontation SF10 No Book vs Lc0 BookX.bin is missing and it would be very interesting to have more data in the comparison.

Yes, I only now realized that it would be a good check. I will do that today.

[Javier Ros]

Yes, I only now realized that it would be a good check. I will do that today.

Thanks

[Laskos]

Yes, I only now realized that it would be a good check. I will do that today.

Thanks

Did it, interesting

1min + 1s TC (Lc0 on RTX 2070, SF10 on 4 i7 threads):


Lc0 No Book vs SF10 No Book (Initial Board position):
Score of lc0_v19_11261 vs SF10: 12 - 6 - 22 [0.575] 40
Elo difference: 52.51 +/- 73.05


Lc0 No Book vs SF10 BookX.bin:
Score of lc0_v191_11261 vs SF10: 5 - 16 - 19 [0.362] 40
Elo difference: -98.07 +/- 79.65


Lc0 BookX.bin vs SF10 No Book:
Score of lc0_v19_11261 vs SF10: 5 - 11 - 24 [0.425] 40
Elo difference: -52.51 +/- 68.93


So, again, the conclusion would be that the variety of openings hurts Lc0. Irrespective whether it is forced into (by a book) or it is not forced (plays all by itself).

i missed the details about the book you're using, but if it's extremely strong, why isn't the conclusion that SF emerges from the opening with a superior position in the majority of games? again, i haven't seen anyone argue NNs surpassed known opening theory.

Now, how do you deal with that? Lc0 "enabled with the unsurpassed opening theory", and having also time on clock advantage because of the book, underperforms by 100 Elo points compared to utterly cherry-picked "Initial Board position", heavily used in the paper.

Maybe we can do an average of the two cases, between diversity by SF10 + Book and diversity by Lc0 + Book, to conclude that "Initial Board position" favors A0(Lc0) by 120 Elo points, and the "12 human openings" by 100 Elo points.

The sole argument would remain that the bias vanishes at very long time control, but all my results combined seem to refute that hypothesis. The bias might diminish somewhat at LTC, that's probably all.

[Michel]

Yes, I only now realized that it would be a good check. I will do that today.

Thanks

Did it, interesting

1min + 1s TC (Lc0 on RTX 2070, SF10 on 4 i7 threads):


Lc0 No Book vs SF10 No Book (Initial Board position):
Score of lc0_v19_11261 vs SF10: 12 - 6 - 22 [0.575] 40
Elo difference: 52.51 +/- 73.05


Lc0 No Book vs SF10 BookX.bin:
Score of lc0_v191_11261 vs SF10: 5 - 16 - 19 [0.362] 40
Elo difference: -98.07 +/- 79.65


Lc0 BookX.bin vs SF10 No Book:
Score of lc0_v19_11261 vs SF10: 5 - 11 - 24 [0.425] 40
Elo difference: -52.51 +/- 68.93


So, again, the conclusion would be that the variety of openings hurts Lc0. Irrespective whether it is forced into (by a book) or it is not forced (plays all by itself).

i missed the details about the book you're using, but if it's extremely strong, why isn't the conclusion that SF emerges from the opening with a superior position in the majority of games? again, i haven't seen anyone argue NNs surpassed known opening theory.

Now, how do you deal with that? Lc0 "enabled with the unsurpassed opening theory", and having also time on clock advantage because of the book, underperforms by 100 Elo points compared to utterly cherry-picked "Initial Board position", heavily used in the paper.

Maybe we can do an average of the two cases, between diversity by SF10 + Book and diversity by Lc0 + Book, to conclude that "Initial Board position" favors A0(Lc0) by 120 Elo points, and the "12 human openings" by 100 Elo points.

The sole argument would remain that the bias vanishes at very long time control, but all my results combined seem to refute that hypothesis. The bias might diminish somewhat at LTC, that's probably all.

I have not completely followed the discussion but I must confess I am puzzled by it.

There is a reliable, time proven method for measuring the strength of chess engines (as practiced by the rating lists). That is a match against a pool of engines using a sufficiently large opening book from which openings are picked at random. There would have been much less room for discussion if this method had also been followed in A0's case.

[Laskos]

Yes, I only now realized that it would be a good check. I will do that today.

Thanks

Did it, interesting

1min + 1s TC (Lc0 on RTX 2070, SF10 on 4 i7 threads):


Lc0 No Book vs SF10 No Book (Initial Board position):
Score of lc0_v19_11261 vs SF10: 12 - 6 - 22 [0.575] 40
Elo difference: 52.51 +/- 73.05


Lc0 No Book vs SF10 BookX.bin:
Score of lc0_v191_11261 vs SF10: 5 - 16 - 19 [0.362] 40
Elo difference: -98.07 +/- 79.65


Lc0 BookX.bin vs SF10 No Book:
Score of lc0_v19_11261 vs SF10: 5 - 11 - 24 [0.425] 40
Elo difference: -52.51 +/- 68.93


So, again, the conclusion would be that the variety of openings hurts Lc0. Irrespective whether it is forced into (by a book) or it is not forced (plays all by itself).

i missed the details about the book you're using, but if it's extremely strong, why isn't the conclusion that SF emerges from the opening with a superior position in the majority of games? again, i haven't seen anyone argue NNs surpassed known opening theory.

Now, how do you deal with that? Lc0 "enabled with the unsurpassed opening theory", and having also time on clock advantage because of the book, underperforms by 100 Elo points compared to utterly cherry-picked "Initial Board position", heavily used in the paper.

Maybe we can do an average of the two cases, between diversity by SF10 + Book and diversity by Lc0 + Book, to conclude that "Initial Board position" favors A0(Lc0) by 120 Elo points, and the "12 human openings" by 100 Elo points.

The sole argument would remain that the bias vanishes at very long time control, but all my results combined seem to refute that hypothesis. The bias might diminish somewhat at LTC, that's probably all.

I have not completely followed the discussion but I must confess I am puzzled by it.

There is a reliable, time proven method for measuring the strength of chess engines (as practiced by the rating lists). That is a match against a pool of engines using a sufficiently large opening book from which openings are picked at random. There would have been much less room for discussion if this method had also been followed in A0's case.

Yes, if that would have been what they reported in the paper, I would not object. But neither the preprint nor the final paper does that. A0 people or cheerleaders seem to disagree on using an opening book, say to fixed depth, as "A0 would have never entered these sorts of openings". Compared to their results, using for example a diversified 8-mover book, Lc0 performs about 100 Elo points weaker than from their picked openings.

They extensively use "Initial Board position" in their paper, even in 1000 games matches. That is silly, but they seem to argue that Chess is the starting position, board and pieces, and A0 learnt that. Aside from the TCEC openings, all the other results are skewed by picked openings, and probably all to favor A0 compared to a normal tester's book.

I actually have short TC result:

Lc0 No Book vs SF10 No Book (Initial Board position):
Score of lc0_v19_11261 vs SF10: 12 - 6 - 22 [0.575] 40
Elo difference: 52.51 +/- 73.05


And from 8-mover diversified PGN book:

Lc0 Book vs SF10 Book
Score of lc0_v19_11261 vs SF10: 7 - 14 - 19 [0.412] 40
Elo difference: -61.43 +/- 79.47

About 100 Elo points worse performance of Lc0 from a "tester's book".

Matthew (one of the authors of the paper) seems to contest the second "normal tester's" result and that A0 strength in the paper is inflated. He suggested that A0 might steer most of the openings its way, so that forcing it to play the general PGN book openings degrades its strength. Also, he suggests that my results are an artifact of short time control used. I tried to show that A0 cannot steer most of the openings the way it likes (say closed positions with few tactics), and it's probably more correct to rate its strength in the pool of regular engines the way the usual engines are rated, that is, with varied openings.