Lizard-NNUE Experiment NOT bad with NNUE Net Evaluation.........

[dkappe]

Done experimenting with this net. For a smoke test, it did pretty well.

Code: Select all

Score of LizardFish1 vs sfdev150720: 22 - 14 - 64  [0.540] 100
Elo difference: 27.85 +/- 40.89

The pgn’s can be found here: https://www.dropbox.com/s/5xpjz9mi78arb2k/lf.pgn?dl=0

[dkappe]

I consider Albert a good friend, but I must disagree a bit. Training a NN to match the eval and search output of a single program seems to be to be a way to clone that program. We might not understand exactly how the NN works compared with say an assembly dump of a programs eval and search functions, but it is a direct attempt to duplicate the program. Training on many sources (programs, human games, self play) is not trying to specifically duplicate another programs search and eval, so I think that wold be allowed. Training for personal use is fine. I am just speaking of training against a program (especially a commercial engine) and then releasing the NN without permission is wrong. I assume testing groups and tournaments would agree, but I would like to hear more opinions.

This is a new world, but the old cloning rules would still apply.

Mark

I should clarify what I know and what I suspect.

1. I know that the nn was trained to approximate Komodo 14 at depth 8.
2. I know that the binary is using Stockfish’s search, not Komodo’s.
3. I don’t know that the nn does a good job of approximating Komodo’s eval at depth 8. It might not be a very good “clone” of Komodo. This can be tested.
4. I suspect but don’t know that with more training the approximation will improve.
5. Neural networks can “generalize,” i.e. produce something greater or better than its inputs.

If the combined search/net doesn’t play like Komodo but like something else, then it’s not a clone but some new thing. Finally some use for Rebel’s punitive clone detector!

[Chessqueen]

I generated training data with Komodo 14 (and a modest amount of python). I used a recent nnue binary to train a net using that data and am running that net using that binary (as are my helpful friends). So this is an approximation of the Komodo 14 eval at depth 8 running on a sf-nnue binary. The name? Nothing serious. It’s a marriage of Komodo and stockfish — LizardFish.

Okay so you created a Hybrid of Komodo and Stockfish, which according to you is stronger than any of them individually

I’ll train it up some more, but I have mixed feelings about distributing a stronger version. It seems almost like a theft of Komodo’s intellectual property. The same sort of cloning (and I think this is much more “cloning” than the usual name calling on this forum) could be done with any uci engine and a modest amount of cpu.

[Milos]

I consider Albert a good friend, but I must disagree a bit. Training a NN to match the eval and search output of a single program seems to be to be a way to clone that program. We might not understand exactly how the NN works compared with say an assembly dump of a programs eval and search functions, but it is a direct attempt to duplicate the program. Training on many sources (programs, human games, self play) is not trying to specifically duplicate another programs search and eval, so I think that wold be allowed. Training for personal use is fine. I am just speaking of training against a program (especially a commercial engine) and then releasing the NN without permission is wrong. I assume testing groups and tournaments would agree, but I would like to hear more opinions.

This is a new world, but the old cloning rules would still apply.

I am sorry but what you write has absolutely zero sense. First, moves played by Komodo as any chess game are public domain. Second playing certain moves in certain positions have nothing to do with cloning Komodo, but training Stockfish eval on moves that are assumed best by Komodo on certain search depth (8). And last, eval trained in this way performs better than eval NN trained at depth 8 of Stockfish only because K at fixed depth 8 is stronger than SF at fixed depth 8.

One thing I really don't understand with those ppl trying to train SF eval. Why are you training on positions produced at fixed depth??? That makes very little to no sense. Just because the original author of SF-NNUE recommended this or because extending learner.cpp to support other parameters than depth is so difficult??? After all, training for fixed number of nodes is also available as a parameter, and even that one would produce better games than fixed depth.
What makes the most sense is to train at fixed time, or even better positions produced by games played with actual TM.

[dkappe]

One thing I really don't understand with those ppl trying to train SF eval. Why are you training on positions produced at fixed depth??? That makes very little to no sense. Just because the original author of SF-NNUE recommended this or because extending learner.cpp to support other parameters than depth is so difficult??? After all, training for fixed number of nodes is also available as a parameter, and even that one would produce better games than fixed depth.
What makes the most sense is to train at fixed time, or even better positions produced by games played with actual TM.

Good questions. I’m certainly able to train with fixed nodes, fixed time, or one of these as a function of the position. But this is literally my first net trained after 1 day of work, so forgive me if I haven’t made all experiments yet.

One open question is whether a relatively small and simple net like this can approximate an eval much deeper than 8. There may be some N (depth, nodes, etc.) above which the net is worse. More experiments.

[Milos]

Good questions. I’m certainly able to train with fixed nodes, fixed time, or one of these as a function of the position. But this is literally my first net trained after 1 day of work, so forgive me if I haven’t made all experiments yet.

One open question is whether a relatively small and simple net like this can approximate an eval much deeper than 8. There may be some N (depth, nodes, etc.) above which the net is worse. More experiments.

I know you are experimenting for a very short period of time, my question was oriented more towards jjoshua and Gekkehenker.

Regarding eval approximation, that's a good question, and I see depth there in a different way.
Actual score at depth 8 search of SF is much more reliable than a small net approximation. On the other hand, depth 8 search takes much more time than NN eval computation. So by replacing handcrafted eval with approximation of depth 8 search one certainly improves quality of eval without sacrificing much and disturbing much the original SF search. OTOH drawback of much higher depth NN approximation might be that it would disturb too much original SF search (for example moves wouldn't be pruned that originally were pruned leading to much fetter tree and eventually weaker search).
Still even if this is true, training at fixed depth makes very little sense, because it effectively leads to quality of NN eval being worse with later game phase. And contrary to handcrafted eval, NN eval takes the same amount of time to be computed despite game phase, while with later game phase handcrafted eval becomes quicker allowing SF to search deeper. Therefore, NN eval should also take progressively higher depth training data as game progresses.

[nabildanial]

I generated training data with Komodo 14 (and a modest amount of python). I used a recent nnue binary to train a net using that data and am running that net using that binary (as are my helpful friends). So this is an approximation of the Komodo 14 eval at depth 8 running on a sf-nnue binary. The name? Nothing serious. It’s a marriage of Komodo and stockfish — LizardFish.

I’ll train it up some more, but I have mixed feelings about distributing a stronger version. It seems almost like a theft of Komodo’s intellectual property. The same sort of cloning (and I think this is much more “cloning” than the usual name calling on this forum) could be done with any uci engine and a modest amount of cpu.

It really is not, since while the NN is trained from games played by Komodo, it is still an NN. If studying Kasparov's games and trying to emulate him makes me his clone then..... my dreams have all come true!!

I am the mysterious tester (this is all dkappe's work), and ran it for 35 games before calling it quits. It was 35 games only (not 1000, sorry), with 30 threads each, for roughly 30+ Million nps for SF10 and 14-15 Million nps for Lizard. I would have played a later version of SF but was told to not be too optimistic, so this was only chosen to try to keep it competitive. A case of underestimating itself if ever one was seen.

Here is the result:

Code: Select all

lizard1-sf10-11-254, Blitz 1.0min+1.0se  2020

                                              12345678901234567890123456789012345
1   SF NNUE halfkp-256 090720,x64 avx2  +113   ½½½½1½½½1½½1½½1½½½1½½½1111½01½1½½½1  23.0/35
2   Stockfish 10 64 POPCNT              -113  ½½½½0½½½0½½0½½0½½½0½½½0000½10½0½½½0   12.0/35

I consider Albert a good friend, but I must disagree a bit. Training a NN to match the eval and search output of a single program seems to be to be a way to clone that program. We might not understand exactly how the NN works compared with say an assembly dump of a programs eval and search functions, but it is a direct attempt to duplicate the program. Training on many sources (programs, human games, self play) is not trying to specifically duplicate another programs search and eval, so I think that wold be allowed. Training for personal use is fine. I am just speaking of training against a program (especially a commercial engine) and then releasing the NN without permission is wrong. I assume testing groups and tournaments would agree, but I would like to hear more opinions.

This is a new world, but the old cloning rules would still apply.

Mark

You won't win any lawsuits if you ever intend to file one for this case, just saying. I know because I'm working as a copyright legal agent right now.

[dkappe]

From the training, the net correctly predicts Komodo’s move choice less than 40% of the time. That could certainly be improved, I imagine, with additional training data. Perhaps calling it an approximation is an overstatement. A loose approximation?

[Tony P.]

It seems almost like a theft of Komodo’s intellectual property.

Fwiw, behavioral cloning (that's a scientific term!), as well as any other kind of knowledge distillation, from Arasan would be perfectly legal And ironically, it might still yield a stronger engine than SF.

[dkappe]

It seems almost like a theft of Komodo’s intellectual property.

Fwiw, behavioral cloning (that's a scientific term!), as well as any other kind of knowledge distillation, from Arasan would be perfectly legal And ironically, it might still yield a stronger engine than SF.

It may be legal (ok, it is legal), but is it right? Right now this net is pretty far away from Komodo in terms of accuracy. But a more highly trained one?