Re: Is this SF NN almost like 20 MB book?
Posted: Tue Aug 04, 2020 11:48 pm
Well thing B having approximate eval of depth 12 or 16 or 18 or whatever stored in 20MBs of data. What would be the score if thing B had 20MB book instead?
Well, do the following test: 1 & 5 you already have, now do 6 & 10, then 11 & 15. See a pattern?Milos wrote: ↑Wed Aug 05, 2020 12:01 amWell thing B having approximate eval of depth 12 or 16 or 18 or whatever stored in 20MBs of data. What would be the score if thing B had 20MB book instead?
All you are actually managed to demonstrate with you dull essay about NNUE analysis is that NNUE is quite a crappy book. Not that it's not.
Difference would actually slightly reduce from 1vs5 to 6vs10 and then it would increase back to original one. But again, this only tells us about the search it tell us nothing about evaluation.dkappe wrote: ↑Wed Aug 05, 2020 12:06 amWell, do the following test: 1 & 5 you already have, now do 6 & 10, then 11 & 15. See a pattern?Milos wrote: ↑Wed Aug 05, 2020 12:01 amWell thing B having approximate eval of depth 12 or 16 or 18 or whatever stored in 20MBs of data. What would be the score if thing B had 20MB book instead?
All you are actually managed to demonstrate with you dull essay about NNUE analysis is that NNUE is quite a crappy book. Not that it's not.
Have you actually run the test, or are you just speculating?Milos wrote: ↑Wed Aug 05, 2020 1:32 amDifference would actually slightly reduce from 1vs5 to 6vs10 and then it would increase back to original one. But again, this only tells us about the search it tell us nothing about evaluation.
Regarding the book, impact is significantly reduced once you go into higher depths. But that is only the case with general books like Cerebellum. With targeted book that is ofc not the case. But my point is that using general book, generated but the engine itself is not much different (fairness-wise) to using interal eval trained by the same engine.
Isn’t that true of literally every evaluation function? Let your evaluation function be “return rand();” and you can use it to generate very crappy opening evals, too. Yet nobody would consider this an opening book.Milos wrote: ↑Wed Aug 05, 2020 12:01 amWell thing B having approximate eval of depth 12 or 16 or 18 or whatever stored in 20MBs of data. What would be the score if thing B had 20MB book instead?
All you actually managed to demonstrate with you dull essay about NNUE analysis is that NNUE is quite a crappy book. Not that it's not.
Superfinal results haven’t been statically significant for a while, it’s on hardware that nobody would consider accessible, the engines constantly update and the results could theoretically be outdated not long after the sufi begins, etc. Getting “valid” superfinal results is incredibly challenging. Attributing that challenge solely to NN opening behavior is absurd. It’s just a drop in the ocean of other issues.Twipply wrote: ↑Tue Aug 04, 2020 11:48 pmI reacted strongly not because of feelings, but because I think this topic has basically invalidated some of the more recent TCEC Superfinal results, and the admins there should stop ignoring it. However, even if my feelings were hurt, that would not invalidate what I've said nor would it validate your post.
Thanks. I'm glad it worked well for you.
The initial part of the game is not quiet. You are conflating two separate suspicions. (The first suspicion probably has a lot more evidence for it than the second.)Dann Corbit wrote: ↑Tue Aug 04, 2020 7:59 pm I suspect that NN approaches work very well for the initial, quiet part of the game.
It's quite the opposite! NNUE isn't learning what opening moves are good. It's learning what moves are good against the openings it plays.
My mistake, I didn't mean to suggest that the "NN=book" idea is my only issue with the validity of the TCEC Superfinals. When I said they're invalid I meant in the sense that if it's not a fair fight then I don't care about the result - not unless the underdog manages to win despite the handicap. Of course, any engine author should realise that the Superfinal results are not likely to be statistically significant, myself included.cucumber wrote: ↑Wed Aug 05, 2020 9:17 amSuperfinal results haven’t been statically significant for a while, it’s on hardware that nobody would consider accessible, the engines constantly update and the results could theoretically be outdated not long after the sufi begins, etc. Getting “valid” superfinal results is incredibly challenging. Attributing that challenge solely to NN opening behavior is absurd. It’s just a drop in the ocean of other issues.