Alphazero news

[jp]

I think that your assumption is wrong at least for stockfish.
The authors of stockfish do not use a big book or a special strong book when they test changes in the code because they assume stockfish will not need to play the opening by itself.

The book that they use is 2moves_v1.pgn that based on my understanding contain 2 random moves by white and black.

They are not specifically testing for opening play & there is no code specific for opening play.

If they were trying to develop its opening play, they would not be using 2moves_v1.pgn.

I do not understand.

2moves_v1.pgn means that stockfish play by itself in the opening in testing so changes that help in the opening stage can be productive.

We can look at the online SF log of suggested changes tested. Many are technical (e.g. pruning, time management, etc.) suggestions. Many are tactical suggestions. There are suggestions for endgame tweaks. I cannot see any that are even vaguely to do with the opening. (But have a look for yourself. I didn't look at every entry, of course, only 2018 ones.)

[yanquis1972]

Have you looked at the best test10 net with perfect2017?

not yet, no, but i hope someone does if i don't get to it. more interested in test30 myself, but provided there's sufficient variation in the book openings (i'll review the games later tonight), so far it's sustaining ~+30 elo against SF w/ book vs without.

i'm curious if, as LR drops, that will decline. instead of honing a particular repertoire leela is still exploring, recently switching from caro-kann to sicilian as preferred defense against 1. e4. i'm assuming this is why A0 had better results vs SF8 when they introduced randomness into the first few moves, but i think that should also be an indicator of good retention.

fwiw, at 5min+5s, test30 is only -80 elo against SF10 at <1:1 ratio (~6MN/s+ for SF10, maybe 5-6kn/s avg for lc0 on my gtx 1080), but i've started adjourning game at move 40 as drawn (+/- 1.50 as win) in an attempt to eliminate endgame losses & produce more games.

[yanquis1972]

After 50 games against each, test30 is -85 elo vs SF10 and -35 against SF+ perfect2017. Think I’ll try running test10 to see if that sticks or if kais trend continues.

[jp]

After 50 games against each, test30 is -85 elo vs SF10 and -35 against SF+ perfect2017.

It's doing worse against bookless SF? What are the error bars?

[yanquis1972]

yes, but in a much smaller sample SF10 has also performed better without the book (vs itself). probably the strength of SF10 from move 1 is underestimated by many, but i also believe the book is intended to produce competitive games, so despite the time edge SF10 typically gets a worse position than it would on its own (pure conjecture on my part).

re the ratings, i'll have to check the early adjournments, as some of them were likely essentially decisive but escaped the win/loss parameters.

pgn links are here (they're extracted from chessbase, so let me know if there's an issue there)

https://www.dropbox.com/s/pedseulix0lxe ... 0.pgn?dl=0
https://www.dropbox.com/s/7sfgxfjpnl3br ... k.pgn?dl=0

[jp]

I think the error margin for only 50 games is too large.

[Rebel]

So, again, the conclusion would be that the variety of openings hurts Lc0.

Makes sense.

[yanquis1972]

I think the error margin for only 50 games is too large.

I plan to run more, but it was pretty consistent, and has been across the board. So far SF10 + test10 (11248) are both outperforming SF10+book, albeit in small samples. (3W 16D 1L for SF10, 2W 3D 0L for id11248 — four draws and a loss against SF10.)

[yanquis1972]

So, again, the conclusion would be that the variety of openings hurts Lc0.

Makes sense.

Variety relative to what? There’s a w/l/d breakdown for A0s games against SF8 in each of the 12 openings (for each color); while it’s clearly a monster with white in many of them it scored significantly worse in others.

If BookX is an 8 move book, it either happens by chance to prefer antileela openings, or the perfect20xx books happen to be pro leela/anti SF. I’m assuming this bc I don’t think a truncated book would hand SF book wins against such a developed net as I worried earlier. Regardless, the perfect books, which are also -8 move and intended for high level engine matches, seem to have a focus on variety and controlled quality, aren’t yielding similar results for me.

[Rebel]

So, again, the conclusion would be that the variety of openings hurts Lc0.

Makes sense.

Variety relative to what? There’s a w/l/d breakdown for A0s games against SF8 in each of the 12 openings (for each color); while it’s clearly a monster with white in many of them it scored significantly worse in others.

If BookX is an 8 move book, it either happens by chance to prefer antileela openings, or the perfect20xx books happen to be pro leela/anti SF. I’m assuming this bc I don’t think a truncated book would hand SF book wins against such a developed net as I worried earlier. Regardless, the perfect books, which are also -8 move and intended for high level engine matches, seem to have a focus on variety and controlled quality, aren’t yielding similar results for me.

I suppose from all the engines AZ and Leela have the best knowledge how to play the opening as after all their learning started from the start position and book variety might move them in somewhat unknown waters. In the end there will come a time they will have to explore the unknown waters also.