TalkChess.com

Laskos wrote: ↑Sun Dec 16, 2018 4:47 pm

Michel wrote: ↑Sun Dec 16, 2018 3:59 pm I have not completely followed the discussion but I must confess I am puzzled by it.

There is a reliable, time proven method for measuring the strength of chess engines (as practiced by the rating lists). That is a match against a pool of engines using a sufficiently large opening book from which openings are picked at random. There would have been much less room for discussion if this method had also been followed in A0's case.

Yes, if that would have been what they reported in the paper, I would not object. But neither the preprint nor the final paper does that. A0 people or cheerleaders seem to disagree on using an opening book, say to fixed depth, as "A0 would have never entered these sorts of openings". Compared to their results, using for example a diversified 8-mover book, Lc0 performs about 100 Elo points weaker than from their picked openings.

They extensively use "Initial Board position" in their paper, even in 1000 games matches. That is silly, but they seem to argue that Chess is the starting position, board and pieces, and A0 learnt that. Aside from the TCEC openings, all the other results are skewed by picked openings, and probably all to favor A0 compared to a normal tester's book.

Matthew (one of the authors of the paper) seems to contest the second "normal tester's" result and that A0 strength in the paper is inflated. He suggested that A0 might steer most of the openings its way, so that forcing it to play the general PGN book openings degrades its strength. I tried to show that A0 cannot steer most of the openings the way it likes (say closed positions with few tactics), and it's probably more correct to rate its strength the way the usual engines are rated, that is, with varied openings, maybe to some depth of 8 or so moves.

Michel, what Kai says. A0 people think what you call the "reliable, time proven method" is "unfair" to A0 because it "forces A0 to play opening moves it does not like". They strangely do not think it is unfair to its opponent to "force A0's opponent to allow A0 to play the opening moves it likes", which is what happens when you have no book. It was also suggested in this thread that it's okay to have a helpless bookless opponent because A0 can always get the positions it wants anyway. That is not true, of course.

jp wrote: ↑Sun Dec 16, 2018 5:19 pmHe suggested that A0 might steer most of the openings its way, so that forcing it to play the general PGN book openings degrades its strength.

I should think that this is indeed true and one doesn't have to be a Chess Genius or AlphaZero to know this.
As far as I can remember from the information I read of the first AlphaZero-Stockfish Match, AlphaZero as White seemed to prefer the English Opening or the Queen Pawn Openings which it often transposes into. So, no Ruy Lopez or Sicilian for Stockfish.
As Black, AlphaZero seemed to prefer the Caro-Kann or the solid Petroff if Stockfish used a KP Opening.
If Stockfish chose a QP Opening as White, again, no problem, as AlphaZero likes QP Openings anyway.
So, again, no Ruy Lopez or Sicilian or any of its countless variations.

shrapnel wrote: ↑Sun Dec 16, 2018 5:57 pm

Laskos wrote:He suggested that A0 might steer most of the openings its way, so that forcing it to play the general PGN book openings degrades its strength.

I should think that this is indeed true ...

shrapnel, you mixed up the quoting function (wrong person).

It is true that A0 likes certain openings. It is not true that it can get everything it likes unless you make its opponents allow it by not giving them a diversified opening book. That's what Kai's testing is about.

yes, the notion that NNs 'steer' positions is observable. Test30 for a bit didn't understand the caro-kann at all (as of last night, it's was its favored defense against e4) and would attempt to transpose it into some sort of french defense (? can't recall details, but always played 3...e6) it 'knew'. similar with test10 playing closed catalan from 3-4 of the the 12 positions. it can be seen in extreme cases with the familiar middle or lategames where it will play objectively worse moves to obtain what it thinks is a winning position.

i still don't see a control (eg how many points, if any, does SF10 lose against SF10 with 'bookx'?)(and could a link be provided, btw? googling turns up nothing for me so far)

personally i'd still advocate for a variety of low-ply (say 1-10 max) openings in testing NNs strength (vs other engines), -- no book for NN, since as stated, that's their point.

intuitively, if someone were to manage to build a 'temp-decay' feature into SF (to introduce some randomness into the first X moves) i think the results would be the most telling, as it introduces variety by forcing the opponent to play what it considers inferior moves. what's bizarre about that is that as i understand it, the same feature used in A0 actually improved its results against SF. i'm curious how that can be if, as we're assuming, it's vital that NNs are given full choice from the first move. the only answer that immediately comes to mind is that, hardware or software, A0 was simply stronger & the variety mean more wins & less draws (/repeat games?)

yanquis1972 wrote: ↑Sun Dec 16, 2018 6:26 pm i still don't see a control (eg how many points, if any, does SF10 lose against SF10 with 'bookx'?)(and could a link be provided, btw? googling turns up nothing for me so far)

If bookless SF lost by x points to SF+bookx, what would you conclude?

As a side note, how confident are we that this wide variance with opening type of results for A0/LC0 is due to their learning the game in some very specific way and not due to, say, MCTS' being substantially weaker in certain types of positions?

One maybe interesting thing to check would be to see if the same pattern is observed for Komodo MCTS.

It's a bit confounded by the fact that we don't know how similar K's MCTS is, but it could be interesting to test; purely from the armchair I find it similarly plausible that this variance is explained by learning something specific as that it is explained by MCTS' being far worse-suited to some position types than others.

I guess the other way to test would be to see how an LC0 with AB search fared across different position types. I don't think there are any forks like this, though. I think there was one a while back that at least had the ability to do minimax backup in the MCTS instead of averaging, which would be closer to traditional search, but I'm not sure if that's been maintained.

Maybe the K MCTS test has already done and I've missed it, in which case mea culpa in advance

MonteCarlo wrote: ↑Sun Dec 16, 2018 6:52 pm As a side note, how confident are we that this wide variance with opening type of results for A0/LC0 is due to their learning the game in some very specific way

In my Opinion, its because A0 is learning the game in a specific way, to avoid complications as far as possible and to avoid anything which distracts it from its single-minded Goal to win the Game, whatever the cost in material.
For instance, in the examples I cited two posts above, AlphaZero would do perfectly well in chess not knowing anything at all about the Ruy Lopez or Sicilian, as there is no way at all that Stockfish can force it to play these Openings, whether as White or Black !

shrapnel wrote: ↑Sun Dec 16, 2018 7:09 pm For instance, in the examples I cited two posts above, AlphaZero would do perfectly well in chess not knowing anything at all about the Ruy Lopez or Sicilian, as there is no way at all that Stockfish can force it to play these Openings, whether as White or Black !

But, though of course it can e.g. avoid playing 1.e4, it does not do perfectly well playing an opponent with a diversified book.

jp wrote: ↑Sun Dec 16, 2018 6:50 pm

yanquis1972 wrote: ↑Sun Dec 16, 2018 6:26 pm i still don't see a control (eg how many points, if any, does SF10 lose against SF10 with 'bookx'?)(and could a link be provided, btw? googling turns up nothing for me so far)

If bookless SF lost by x points to SF+bookx, what would you conclude?

depending on what 'x' is, that the book(or resultant TC advantage) is overly strong. as i've said, there's a massive difference between opening variety & simply outbooking the opponent. if you want to test the former it's vital to make sure you're not falling into the latter. so low-ply & with relatively high randomness as to moves selected, imo, is sensible.

i do think that with longer TCs, as uri pointed out earlier, SF10 will actually improve more (relative to Lc0) vs a strong book. i don't really understand the LTC argument w/r/t lc0, provided the book is sufficiently shallow. a thoroughly trained NN has what is essentially an opening repertoire, so i agree it's essential to test it against a large variety. (much larger than deepmind did; as i said, i believe those openings are primarily important for benchmarking against A0)

yanquis1972 wrote: ↑Sun Dec 16, 2018 7:54 pm depending on what 'x' is, that the book(or resultant TC advantage) is overly strong. as i've said, there's a massive difference between opening variety & simply outbooking the opponent. if you want to test the former it's vital to make sure you're not falling into the latter. so low-ply & with relatively high randomness as to moves selected, imo, is sensible.

Yeah, I'm not sure if we can make a good guess for what 'x' might be, but there's no way that SF+bookX is simply outbooking its opponent. If bookX were so great an advantage, Lc0+bookX would not have done so badly against SF no book, even if Lc0 did not like those openings.

Maybe some info on how many plies bookX goes would be good.

yanquis, if Kai gives the url to download bookX, could you run the test you want to see?

yanquis1972 wrote: ↑Sun Dec 16, 2018 7:54 pm i don't really understand the LTC argument w/r/t lc0, provided the book is sufficiently shallow. a thoroughly trained NN has what is essentially an opening repertoire, so i agree it's essential to test it against a large variety. (much larger than deepmind did; as i said, i believe those openings are primarily important for benchmarking against A0)

Yeah, you could say that the NN engine is essentially outbooking its bookless opponent. The Deepmind argument for not giving its opponent a book is that neither has a book, but no one here is disagreeing that the NN engine obviously has an opening repertoire.