Asking to people who believe Leela NN is a book, what they think about SF NN now?

[the_real_greco]

[/quote]

By this definition, are not the evaluation constants within an ordinary hand-tuned evalution also and opening book?
[/quote]

(I'm assuming you're talking about the is-of-predication definition?)

I guess I'm confused why this should be? Hand-tuned evaluations return centipawn values, not moves. NN policy functions return... probabilities, I guess, but those correspond to moves very closely.

It's just hard (for me at least) to see how a function from positions to moves is (of predication) not a book, even if it is (of identity) not a book.

[carldaman]

For a long time I wanted to see a smart book feature, where the engine would look at its book as a starting point, but keep searching, and if it found something stronger by a certain threshold amount it would override the book. I guess that's a little bit like what we ended up having with Leela.

[corres]

Opposite to Leela net in the net of SF+NNUE there is no "policy-head", so SF+NNUE search for the position of max value (as AB engine also do it) and it does not know (from learning games) to where it would move (as it is obvious in the case of an opening book). So the playing of the Leela is similar to as if she would use book, but playing NNUE is similar to playing of Classical Stockfish.
Note
I think Classical Stockfish would reach the present day Elo of SF+NNUE after a years development.
Naturally SF+NNUE also would yield more Elo, so the Elo difference maybe will not lessen (drastically)...

[Ovyron]

A point Alayan is trying to make (IMO in a pretty clear way) is that NN has some memory about trained positions.

Can you go and show us where this resides? Can anybody tell us what parameters are responsible of memorizing the best moves in some positions? Is it even possible to remove parts of the NN so a memorized position is "forgotten", and the NN has to come up with something for it "from scratch" while the rest of positions are still "in book"?

No.

In any book, you can alter what it has memorized about the opening position, mark all moves red or delete them, and then the book will have 0 content on it, and the engine will have to think of a move by itself, but will be able to pick up on the second move everything the book has memorized.

That NN is completely different, and this is why, no matter how much control you have over its contents, or even if you train one from scratch and omit this position, you can't achieve the same effect.

Because the main difference between a book and a NN is that the book looks at the current position, checks what it has memorized, and outputs a move.

The NN only looks at the current position to generate more positions, to look at their patterns, and to measure how likely those patterns are to win, draw or lose, and then it produces a list of moves with the best probabilities. And it continues to spend its time generating more positions, and repeating this process, to get a more accurate move ordering.

That's why, no matter how many "nodes" or "depth" it has reached, it could always switch a move into first position and play it at the last moment, as the latest patterns it has seen has made it better than what it appeared as best up until this point. When have you seen a book do that?

That NN and a book are so different that the book requires a dice to come with it. Or that the GUI using the book comes with a dice. Or that the engine using the book comes with a dice.

The book says that there's some probability that 1.e4 is played, and another that 1.d4 is played. How does it know how to play? It produces a random number, it takes a look at it and decides, and if you play enough games over the long run it'll have played 1.e4 and 1.d4 in a way that approaches this probability (despite not keeping a story of what has been played. So if their probability is 50%, the random number is meant to make how often those are played close to equal, without knowing how often they've been played before.)

More than anything, a book is a move blocker. It blocks moves marked red, or absent from the book, from being picked by the random dice throw. That NN is not random, and never blocks any move, all of them could get to the top at any time, that's why it can play moves never seen before in openings that no book ever considered, and blow everyone away.

Now that I've explained how that NN works and how a book works, can you see how they're nothing alike?

[Alayan]

Can you go and show us where this resides? Can anybody tell us what parameters are responsible of memorizing the best moves in some positions? Is it even possible to remove parts of the NN so a memorized position is "forgotten", and the NN has to come up with something for it "from scratch" while the rest of positions are still "in book"?

No.

Can you go and show us the neurons responsible in a human chess player for knowing that 1. e4 and 1. d4 are good.

No ?

Therefore by your logic humans can't possibly have memorized these moves. But we know humans do memorize, so your logic is plain wrong.

The rest of your post about how a NN works (I skimmed through it) is therefore completely irrelevant. It's not proving at all that no memorization occurs.

[Ovyron]

Can you go and show us the neurons responsible in a human chess player for knowing that 1. e4 and 1. d4 are good.

No ?

For a given human, you could check what parts of the brain are activated when they're on the opening position of chess, which ones are illuminated when 1. e4 or 1. d4 are being considered.

Scientists used to think that memory resided in some specific part of the brain, and wanted to know where, but these tests proved that memories are stored all around the brain. Different zones of the brain get activated when retrieving memories. There's specific parts of the brain responsible for retrieving them, and if they're damaged they can't be accessed, but they're not where they're stored.

So different people have 1.e4 and 1.d4 memorized in different parts of the brain.

Do you know what's funny? Hypnosis. An experienced hypnotist can make you forget that, say, the number 5 exists. So when counting you'd skip from 4 to 6. And then she could make you count all your fingers of your hands and arrive at 11. And you'd not even know why you're counting 11 fingers. You're not just forgetting 5 existed, you aren't aware that something is missing at all.

In the same way, a hypnotist could erase 1.e4 and 1.d4 as possible moves from your memory, and every time you played from the white side, you'd not even consider those moves, and you'd not even know there's some moves missing from your consideration.

Can you replicate this memory blockage on a NN?

No.

Because whenever an NN checks the opening position, it's the first time it sees it, to make 1.e4 and 1.d4 reach the top it has to do it from scratch. Only, it's done extremely intelligently and fast, just after 1 position generated it'll recognize the patterns that it has learned and that can suffice to bring them to the top, but it has nothing to do with the opening position, as if you switch pieces around just after 1 position it'll also come up with a decent move already, and the only reason it's not as good it's because the person training it hasn't shown it the patterns that appear from this position with pieces switched around.

If you want to know more about the difference, research memory against pattern recognition, which are entirely different processes.

That NN is doing artificial pattern recognition and THAT'S EVEN BETTER THAN A BOOK. A book enslaves the engine to be forced to play whatever it has in it, just like opening suites, or whatever humans have decided must be used to widen variety, while that NN can still discover new stuff missing from any book, as it generates positions and discovers new patterns that are outside everyone's horizon.

The only reason NNs don't perform as well is the time taken for this, it's not about move quality because the time gained with a book will compensate. It'd be interesting to hold a no book match, where that NN gets the normal game time to make its moves against the entity with the book. However, when the book is over, the clocks are reset and the NN regains her time, making it fair.

Chances are that NN will perform better, specially considering books are full of poor generic openings meant to increase variety without regard to quality, because they'll just play with reversed colors, in my proposal the NN will never be forced to play those poor moves from the other side, and could show NNs > Books (like in humans where learning many opening patterns and recognizing them is better than just memorizing some openings.)

[AndrewGrant]

Because whenever an NN checks the opening position, it's the first time it sees it, to make 1.e4 and 1.d4 reach the top it has to do it from scratch. Only, it's done extremely intelligently and fast, just after 1 position generated it'll recognize the patterns that it has learned and that can suffice to bring them to the top, but it has nothing to do with the opening position, as if you switch pieces around just after 1 position it'll also come up with a decent move already, and the only reason it's not as good it's because the person training it hasn't shown it the patterns that appear from this position with pieces switched around.

I'm behind NN research by 4 decades and even I'm aware that its a well documented phenomena that NNs bake in a memorization of the dataset.

https://arxiv.org/pdf/1611.03530.pdf Read that, and if you still don't think its possible for Leela to "memorize" a selection of openings, please email the authors at the top of that document, as they will be far less generous than users here.

[Ovyron]

Read that, and if you still don't think its possible for Leela to "memorize" a selection of openings, please email the authors at the top of that document, as they will be far less generous than users here.

Fine. Leela can "memorize" a selection of openings. But she can't memorize a selection of openings. A book can memorize them, because a book is nothing but that memorization.

That's why Leela behaves the same even on new positions she hasn't seen before, if they reach similar patterns, because she has "memorized" those patterns from trained data.

If it could memorize them it wouldn't suffer from playing her opening moves instantly from memory, but you can make her play at 1 node the opening (so she plays instantly from memory) and the rest at full strength to see the difference between "memorization" and memorization, which is immense.

If it was the same thing, Leela would play stronger by playing from what she has memorized instantly and save that time to play stronger the rest of the game.

[AndrewGrant]

Read that, and if you still don't think its possible for Leela to "memorize" a selection of openings, please email the authors at the top of that document, as they will be far less generous than users here.

Fine. Leela can "memorize" a selection of openings. But she can't memorize a selection of openings. A book can memorize them, because a book is nothing but that memorization.

You've gone from arguing the idea, to arguing the semantics. So I think we are done here.

[dkappe]

Somehow a leela style net that never saw positions with more than 18 pieces was able to “memorize” opening moves. Yet somehow people arguing this tired old argument are unable to memorize this refutation.