Lizard-NNUE Experiment NOT bad with NNUE Net Evaluation.........

[Daniel Shawul]

Many AB engines use Texel tuning with data scored by Stockfish at depth=8. Remember the popular quiet.epd and the 3 million
fast SF games generated by Alvaro for the purpose of eval tuning ?
I do not consider this to be a clone nor lizard-nnue that mimics Komodo at depth=8.

[Raphexon]

Many AB engines use Texel tuning with data scored by Stockfish at depth=8. Remember the popular quiet.epd and the 3 million
fast SF games generated by Alvaro for the purpose of eval tuning ?
I do not consider this to be a clone nor lizard-nnue that mimics Komodo at depth=8.

Calling it a clone would be akin to calling a net trained on grandmaster games to be a human clone.

[Milos]

Many AB engines use Texel tuning with data scored by Stockfish at depth=8. Remember the popular quiet.epd and the 3 million
fast SF games generated by Alvaro for the purpose of eval tuning ?
I do not consider this to be a clone nor lizard-nnue that mimics Komodo at depth=8.

Calling it a clone would be akin to calling a net trained on grandmaster games to be a human clone.

Or a painting created by an artist a clone of paint tubes .

[Tony P.]

Well, maybe it's a good idea to investigate what we are talking about, without the punitive, as Mark said NN's are a whole different world. A quick test with some adviced nets. Using a random epd of 10,000 positions at depth=1.

Thanks for the revealing analysis

What's the correlation between AllieStein and T60-4300? I'm wondering if the replacement of Leela's search by Allie's makes enough difference.

[chrisw]

I generated training data with Komodo 14 (and a modest amount of python). I used a recent nnue binary to train a net using that data and am running that net using that binary (as are my helpful friends). So this is an approximation of the Komodo 14 eval at depth 8 running on a sf-nnue binary. The name? Nothing serious. It’s a marriage of Komodo and stockfish — LizardFish.

I’ll train it up some more, but I have mixed feelings about distributing a stronger version. It seems almost like a theft of Komodo’s intellectual property. The same sort of cloning (and I think this is much more “cloning” than the usual name calling on this forum) could be done with any uci engine and a modest amount of cpu.

It really is not, since while the NN is trained from games played by Komodo, it is still an NN. If studying Kasparov's games and trying to emulate him makes me his clone then..... my dreams have all come true!!

I am the mysterious tester (this is all dkappe's work), and ran it for 35 games before calling it quits. It was 35 games only (not 1000, sorry), with 30 threads each, for roughly 30+ Million nps for SF10 and 14-15 Million nps for Lizard. I would have played a later version of SF but was told to not be too optimistic, so this was only chosen to try to keep it competitive. A case of underestimating itself if ever one was seen.

Here is the result:

Code: Select all

lizard1-sf10-11-254, Blitz 1.0min+1.0se  2020

                                              12345678901234567890123456789012345
1   SF NNUE halfkp-256 090720,x64 avx2  +113   ½½½½1½½½1½½1½½1½½½1½½½1111½01½1½½½1  23.0/35
2   Stockfish 10 64 POPCNT              -113  ½½½½0½½½0½½0½½0½½½0½½½0000½10½0½½½0   12.0/35

I consider Albert a good friend, but I must disagree a bit. Training a NN to match the eval and search output of a single program seems to be to be a way to clone that program. We might not understand exactly how the NN works compared with say an assembly dump of a programs eval and search functions, but it is a direct attempt to duplicate the program. Training on many sources (programs, human games, self play) is not trying to specifically duplicate another programs search and eval, so I think that wold be allowed. Training for personal use is fine. I am just speaking of training against a program (especially a commercial engine) and then releasing the NN without permission is wrong. I assume testing groups and tournaments would agree, but I would like to hear more opinions.

This is a new world, but the old cloning rules would still apply.

Mark

You won't win any lawsuits if you ever intend to file one for this case, just saying. I know because I'm working as a copyright legal agent right now.

There is so little money in computer chess that any lawsuits are just a waste of time and money. I was not speaking about law, I was speaking about what is right.

Well, since you're entirely wrong, that's a misuse of the word "right".

Training a NN on one single programs games is an attempt to duplicate that programs eval and search.

It will by definition never duplicate it, it might approximate it, but that's not the same thing at all. Misuse of the word "duplicate".

It is a form of cloning.

Debatable. "form of" is a weasel expression, objective get the bad word cloning into association. Misuse of language.

That is the whole point of NNs afterall, to be trained to be like something else.

Debatable. NNs have many other purposes, and the purpose here is to make something improved of the input data. Again, wrong.

It is a derivative work,

A Derivative work is a work that contains substantial copyright elements of another. Clearly that is not the case here, it contains zero copyright elements of another. Misuse of the wprd derivative.

and online tournaments should ban them from competing

Yeah, right, when all else fails we'll get a posse going. Been here before I believe?

unless the original programs authors agree.

Shucks. Once you sold your program you don't own or control its output. The ability to have it perform producing unknown outputs in different situations is a right of new owner or licensor. Nice to know you want people to pay for your program but you still want to control what they do with its output. Well, tough, the output games are property the new user, until he publishes them in which case they automatically are copyright exempt.

TCEC recently decided Komodo MCTS could not play in the same tournaments as regular Komodo since K MCTS mostly shared the eval in regular Komodo.

Let use SFNNUE as an example. It was trained on Stockfish games. This is a huge advantage over the lc0 approach of self play training. Much of the strength of SFNNUE has to be due to the eval and search in Stockfish. If SF NNUE ends up being stronger than Stockfish (as some report) then that would tend to discourage people from improving regular Stockfish. And is it fair for SFNNUE be in tournaments against regular Stockfish? Maybe they are fine with it. I do not know. But I would not be fine with SFNNUE using a Komodo trained NN without our permission and competing against any Komodo version.

Tough. You already sold that right in exchange for money.

I would like to see everyone working together, with permission,

Your permission is not needed, lawfully, ethically of any other reason, you sold the rights to use the program for money, too late to turn round and complain. Nothing to stop you making some project, joint of otherwise, of your own or course.

Let us know if you find a way to tie your customers up in EULA knots to prevent them doing that they want with games output. I doubt any attempt to control use of output chess games could possibly find any lawful basis. Would you like to buy my program but you can't use actually it?

to make the best we can. I think it is quite possible a Komodo version using Komodo trained NNUEs could be better than regular Komodo. And if a repeated cycle of playing games with that new program then retraining it could lead to even more gains.

Mark

[Tony P.]

No matter the legal status of a net distilled from Komodo's output without Mark's permission, what he can do is have it banned from C3. If the trainers are fine with the loss of this avenue for promotion, they can go ahead with the distillation

[Milos]

No matter the legal status of a net distilled from Komodo's output without Mark's permission, what he can do is have it banned from CCC. If the trainers are fine with the loss of this avenue for promotion, they can go ahead with the distillation

Banned based on what? Have you even read the charter?

[Tony P.]

Oops, I meant chess.com's Computer Chess Championship (C3 is a better abbreviation for it, right?). There are no grounds for a ban from Computer Chess Club, of course.

[Raphexon]

Also training from Komodo games won't approximate Komodo's eval and it certainly won't approximate Komodo's search.
NNs don't just copy.

Even if a Houdart would reverse-engineered* Komodo's search and add an NNUE trained on Komodo games it wouldn't play like Komodo. Although it probably does play in a very anti-Komodo way.
If Komodo has certain flaws in its eval, then the NN will exploit them hard.
But that's not approximating Komodo's eval.

*Just to be clear, I 100% disapprove of reverse engineering copyrighted material.
But using game results as training isn't reverse engineering.

[kranium]

I would like to see everyone working together, with permission, to make the best we can.

Mark

Many people are contributing to NUUE
but Komodo is commercial...

I hope you'll reconsider your objections, as using Komodo eval and/or games might prove helpful/interesting.

Larry is posting some really nice testing...
I'd love to see you get involved.