Lizard-NNUE Experiment NOT bad with NNUE Net Evaluation.........

[smatovic]

Dietrich Kappe, Milos and Albert Silver talking about Copyright and NNs, yea sure.

Mark said clearly what is OK for him and what not.

What happens next if you do not respect this? Update of Komodo EULA maybe? It is up to you.

--
Srdja

*** edit ***

as an thought experiment...I take all games from Karpov, make up an NN engine
which plays 100% Karpov style, advertise this as Karpov game trained engine,
without any deal to make with Karpov? Or another thought experiment, I take
a living Shakespear, use all his writings to create an NN which comes up with
100% Shakespear style, all for free? Without any restrictions to me as NN
*trainer*? This is all new stuff, how to deal with this is work in progress,imo.

[Tony P.]

^^ Generally, one can ask the author(s) of a program if they mind its cloning. There are strong indications that Jon wouldn't mind (though you'd still better ask):

1. His very choice of the MIT license.
2. The post where he explained his attitude to the reuse of his software.
3. The existence of Arasan Programmer's Guide and especially this sentence:

While no formal support is offered for this software, if you do find bugs in it, or discover a way to improve it, I would like to hear from you.

Moreover, if a net trained by Arasan excels its teacher, Jon will still be regarded as the project founder who's built an engine with ample positional knowledge that will have been instrumental in the bootstrap of the net.

Perhaps the process can even be iterated to great effect, i.e., after inserting the net into Arasan and retuning its search parameters, the output of the hybrid's AB search at some depth may train an even preciser net, and so on until convergence

Speaking of the Komodo case, the verdict on the copyright to its output is not so clear a priori, e.g. there was the Torah Soft Ltd. v. Drosnin case where matrices output by the program were ruled copyrightable because '[c]reating a matrix is unlike the creative process used in many computer art programs, which permit an end-user to create an original work of art in an electronic medium'. This doesn't imply that Komodo cloners would face a lawsuit, but the risk seems to exist.

[Rebel]

If the combined search/net doesn’t play like Komodo but like something else, then it’s not a clone but some new thing. Finally some use for Rebel’s punitive clone detector!

Well, maybe it's a good idea to investigate what we are talking about, without the punitive, as Mark said NN's are a whole different world. A quick test with some adviced nets. Using a random epd of 10,000 positions at depth=1.

Code: Select all

Positions 10000 Leele NN-64 SF-NN SF11  T40-1 T60-4
Leelentein-14   ----- 73.30 43.24 38.92 77.35 73.30
NN-64194        73.30 ----- 43.17 38.83 72.31 75.97
SF-NN           43.24 43.17 ----- 45.72 42.43 42.83
SF11            38.92 38.83 45.72 ----- 38.11 38.27
T40-1541        77.35 72.31 42.43 38.11 ----- 71.45
T60-4300        73.30 75.97 42.83 38.27 71.45 -----

 1.         Engine Leelentein-14 and T40-1541 show a similarity of 77.35%
 2.         Engine NN-64194 and T60-4300 show a similarity of 75.97%
 3.         Engine Leelentein-14 and NN-64194 show a similarity of 73.30%
 4.         Engine Leelentein-14 and T60-4300 show a similarity of 73.30%
 5.         Engine NN-64194 and T40-1541 show a similarity of 72.31%
 6.         Engine T40-1541 and T60-4300 show a similarity of 71.45%

Quite some high similarities between the nets except the new approach with the 30Mb NN.BIN net from SF-NN version 0.1

Comments?

[JohnS]

I generated training data with Komodo 14 (and a modest amount of python). I used a recent nnue binary to train a net using that data and am running that net using that binary (as are my helpful friends). So this is an approximation of the Komodo 14 eval at depth 8 running on a sf-nnue binary. The name? Nothing serious. It’s a marriage of Komodo and stockfish — LizardFish.

I’ll train it up some more, but I have mixed feelings about distributing a stronger version. It seems almost like a theft of Komodo’s intellectual property. The same sort of cloning (and I think this is much more “cloning” than the usual name calling on this forum) could be done with any uci engine and a modest amount of cpu.

How much computing time did all this take and what computer did you use thanks.

[Chessqueen]

^^ Generally, one can ask the author(s) of a program if they mind its cloning. There are strong indications that Jon wouldn't mind (though you'd still better ask):

1. His very choice of the MIT license.
2. The post where he explained his attitude to the reuse of his software.
3. The existence of Arasan Programmer's Guide and especially this sentence:

While no formal support is offered for this software, if you do find bugs in it, or discover a way to improve it, I would like to hear from you.

Moreover, if a net trained by Arasan excels its teacher, Jon will still be regarded as the project founder who's built an engine with ample positional knowledge that will have been instrumental in the bootstrap of the net.

Perhaps the process can even be iterated to great effect, i.e., after inserting the net into Arasan and retuning its search parameters, the output of the hybrid's AB search at some depth may train an even preciser net, and so on until convergence

Speaking of the Komodo case, the verdict on the copyright to its output is not so clear a priori, e.g. there was the Torah Soft Ltd. v. Drosnin case where matrices output by the program were ruled copyright table because '[c]reating a matrix is unlike the creative process used in many computer art programs, which permit an end-user to create an original work of art in an electronic medium'. This doesn't imply that Komodo cloners would face a lawsuit, but the risk seems to exist.

Some of you were not probably around, but this is no diffent then when people bought Chessmaster 3000 and gave it different personalities. What I am trying to say here is that in order to give CM personalities or in this case of Komodo people will still have to purchase Komodo if they do NOT have it already before they can start using NNUE, as a matter of fact just like in the case of ChessMaster the sale of ChessMaster increased since people like to experiment to make it better. It is NOT like he is giving LizardFish1 for Free to the Public on a certain Website, this more like advertising for Komodo Saying hey all of you with Komodo 14 look what you can do, you can create a Hybrid cross between Komodo + Stockfish using a NNUE Net.

[Alayan]

I consider Albert a good friend, but I must disagree a bit. Training a NN to match the eval and search output of a single program seems to be to be a way to clone that program.

I am just speaking of training against a program (especially a commercial engine) and then releasing the NN without permission is wrong. I assume testing groups and tournaments would agree, but I would like to hear more opinions.

This is a new world, but the old cloning rules would still apply.

Mark

I disagree.

Games are not copyrightable, hence if you own Komodo and have it play games, the Komodo authors have no copyright on those games and you are free to use them for something like training a NN. The case for copyrighting a position's evaluation is barely better.

More importantly, the end result isn't anything like the real Komodo. It's not using any part of it, and isn't playing like it.

And it's not like using Komodo to train a net is more than a gimmick. One could probably use Minic depth 10 for training and still obtain something very sensible, propelled by Stockfish's search, though it would be less efficient computationally than just going with SF searches.

[Chessqueen]

I consider Albert a good friend, but I must disagree a bit. Training a NN to match the eval and search output of a single program seems to be to be a way to clone that program.

I am just speaking of training against a program (especially a commercial engine) and then releasing the NN without permission is wrong. I assume testing groups and tournaments would agree, but I would like to hear more opinions.

This is a new world, but the old cloning rules would still apply.

Mark

I disagree.

Games are not copyrightable, hence if you own Komodo and have it play games, the Komodo authors have no copyright on those games and you are free to use them for something like training a NN. The case for copyrighting a position's evaluation is barely better.

More importantly, the end result isn't anything like the real Komodo. It's not using any part of it, and isn't playing like it.

And it's not like using Komodo to train a net is more than a gimmick. One could probably use Minic depth 10 for training and still obtain something very sensible, propelled by Stockfish's search, though it would be less efficient computationally than just going with SF searches.

You are correct and it is no different than when we experimented with ChessMaster 3000. Some of you were not probably around, but this is no diffent then when people bought Chessmaster 3000 and gave it different personalities. What I am trying to say here is that in order to give CM personalities or in this case of Komodo people will still have to purchase Komodo if they do NOT have it already before they can start using NNUE, as a matter of fact just like in the case of ChessMaster the sale of ChessMaster increased since people like to experiment to make it better. It is NOT like he is giving LizardFish1 for Free to the Public on a certain Website, this is more like advertising for Komodo Saying hey all of you with Komodo 14 look what you can do, you can create a Hybrid cross between Komodo + Stockfish using a NNUE Net.

[mjlef]

I generated training data with Komodo 14 (and a modest amount of python). I used a recent nnue binary to train a net using that data and am running that net using that binary (as are my helpful friends). So this is an approximation of the Komodo 14 eval at depth 8 running on a sf-nnue binary. The name? Nothing serious. It’s a marriage of Komodo and stockfish — LizardFish.

I’ll train it up some more, but I have mixed feelings about distributing a stronger version. It seems almost like a theft of Komodo’s intellectual property. The same sort of cloning (and I think this is much more “cloning” than the usual name calling on this forum) could be done with any uci engine and a modest amount of cpu.

It really is not, since while the NN is trained from games played by Komodo, it is still an NN. If studying Kasparov's games and trying to emulate him makes me his clone then..... my dreams have all come true!!

I am the mysterious tester (this is all dkappe's work), and ran it for 35 games before calling it quits. It was 35 games only (not 1000, sorry), with 30 threads each, for roughly 30+ Million nps for SF10 and 14-15 Million nps for Lizard. I would have played a later version of SF but was told to not be too optimistic, so this was only chosen to try to keep it competitive. A case of underestimating itself if ever one was seen.

Here is the result:

Code: Select all

lizard1-sf10-11-254, Blitz 1.0min+1.0se  2020

                                              12345678901234567890123456789012345
1   SF NNUE halfkp-256 090720,x64 avx2  +113   ½½½½1½½½1½½1½½1½½½1½½½1111½01½1½½½1  23.0/35
2   Stockfish 10 64 POPCNT              -113  ½½½½0½½½0½½0½½0½½½0½½½0000½10½0½½½0   12.0/35

I consider Albert a good friend, but I must disagree a bit. Training a NN to match the eval and search output of a single program seems to be to be a way to clone that program. We might not understand exactly how the NN works compared with say an assembly dump of a programs eval and search functions, but it is a direct attempt to duplicate the program. Training on many sources (programs, human games, self play) is not trying to specifically duplicate another programs search and eval, so I think that wold be allowed. Training for personal use is fine. I am just speaking of training against a program (especially a commercial engine) and then releasing the NN without permission is wrong. I assume testing groups and tournaments would agree, but I would like to hear more opinions.

This is a new world, but the old cloning rules would still apply.

Mark

You won't win any lawsuits if you ever intend to file one for this case, just saying. I know because I'm working as a copyright legal agent right now.

There is so little money in computer chess that any lawsuits are just a waste of time and money. I was not speaking about law, I was speaking about what is right. Training a NN on one single programs games is an attempt to duplicate that programs eval and search. It is a form of cloning. That is the whole point of NNs afterall, to be trained to be like something else. It is a derivative work, and online tournaments should ban them from competing unless the original programs authors agree. TCEC recently decided Komodo MCTS could not play in the same tournaments as regular Komodo since K MCTS mostly shared the eval in regular Komodo.

Let use SFNNUE as an example. It was trained on Stockfish games. This is a huge advantage over the lc0 approach of self play training. Much of the strength of SFNNUE has to be due to the eval and search in Stockfish. If SF NNUE ends up being stronger than Stockfish (as some report) then that would tend to discourage people from improving regular Stockfish. And is it fair for SFNNUE be in tournaments against regular Stockfish? Maybe they are fine with it. I do not know. But I would not be fine with SFNNUE using a Komodo trained NN without our permission and competing against any Komodo version.

I would like to see everyone working together, with permission, to make the best we can. I think it is quite possible a Komodo version using Komodo trained NNUEs could be better than regular Komodo. And if a repeated cycle of playing games with that new program then retraining it could lead to even more gains.

Mark

[jhellis3]

If what people are after is relatively strong low depth play, they are more than welcome to use Crystal. Free of charge....

[Milos]

If what people are after is relatively strong low depth play, they are more than welcome to use Crystal. Free of charge....

How is Crystal low depth search different than SF's? Is it mainly move ordering and no pruning in PV or differences are more substantial?