Is Allie a 'Leela net player'?

[Raphexon]

There are actually four levels of originality. There is completely original engines, as in search and evaluation are built from the ground up, or the search and evaluation has changed so much from an existing engine as to have no resemblance to the original engine anymore. There are neural network players, which are engines whose search is built from the ground up but whose evaluation is entirely a neural network architecture from an existing engine, that could be considered as a library. There are derivatives, as in the search and evaluation both are derived from an existing engine with enough changes to not be considered a clone, but not enough changes to be considered original. And then there are clones, which are engines almost exactly the same as an existing engine.

Leela-type nets feature a "policy head" that gives exploration weights to each possible move in a position. It plays a major role into the strength of Leela, as it's guiding the PUCT search and its pruning.

A strong engine that is using Leela-type nets MUST make use of this policy information.

The end result is that while Allie's search code has been written by gonzo and has some unique properties, it behaves largely the same as Leela because at the root it's PUCT guided by the net's policy head, even if Allie favors higher score over top visits and has some light min-maxing.

It's an example of how the use of a similar architecture in one place (eval) can enforce more similarity elsewhere (search).

+Allie's net is trained on T60 games (among others) and before some recent updates Allie's eval almost always mimicked Leela's eval. (Just inflated massively)

If you replace SF's multithreading with ABDADA (or YBWC) and train a net a little differently you quickly have something that's just as *unique* as Allie.

[dkappe]

+Allie's net is trained on T60 games (among others) and before some recent updates Allie's eval almost always mimicked Leela's eval. (Just inflated massively)

If you replace SF's multithreading with ABDADA (or YBWC) and train a net a little differently you quickly have something that's just as *unique* as Allie.

A0lite ‘mimics’ leela’s eval because they both ‘mimic’ A0’s eval (read the papers). You’ve correctly identified that there is a parent engine, you’ve just identified the wrong one (and if you claim that a0lite’s 95 line python implementation of mcts/uct is a clone of leela, I have a bridge to sell you).

[xr_a_y]

[...]

TCEC bans and the community as a whole disfavours clones and derivatives, but both are currently unclear about the status of neural network players, and both are treating the ankan NN engines (Allie/Leela) and the NNUE engines (Stockfish/Igel/Rubichess/etc) differently when they should be treated the same. I would be satisfied if either Igel/Rubichess/Minic/etc was treated like Allie or vice versa in their relationship with Stockfish and Leela respectively.

There are, at least, in my opinion, 3 things to take into account in the NNUE question (not speaking about Allie/Leela here):

1°) use of the NNUE technology itself : can be near a copy-paste (like in Minic for instance), but can be an own interpretation (even supporting the same architecture)
2°) use of data for training. Where are the data from ? Own engine ? many engines ? Stockfish genfen only ?
3°) kind of learner used, and possible dependency to Stockfish (q)search. If Nodchip learner is used as-it, then a part of Stockfish (q)search is used.

Using, for instance, both SF data and Nodchip SF learner (thus qsearch in learning phase) might be considered too much of a "copy".
Using, own data and SF learner might be considered borderline.
Using, own data, and own learner, might be OK if NNUE technology (and subsequent copy-pasting) is indeed view as a "lib".

Still, only my own opinions here !

Here is two things that can contribute to make thing clearer

1°) make the NNUE evaluation code indeed a library : I don't think this is that hard, and this will not limit too much the possibility to investigate new architecture. But this probably requiere to build a common "position" object from which to extract needed input with a common API and without overhead (this is very possible, at least for bitboards based engines)
2°) be able to use external tool to build nets (thus not depending on SF qsearch and probably gain speed in the process being able to rely on big framework and their GPU capabilities). This may be harder but there are solutions. For instance, only use quiet positions for training, or build a simple commun qsearch function outside the engine.

[Madeleine Birchfield]

Using, for instance, both SF data and Nodchip SF learner (thus qsearch in learning phase) might be considered too much of a "copy".
Using, own data and SF learner might be considered borderline.
Using, own data, and own learner, might be OK if NNUE technology (and subsequent copy-pasting) is indeed view as a "lib".

There is a fourth option, namely using SF data and own trainer. This is the approach that Allie currently uses when it uses the Stein network in TCEC. The trainers for Leela nets according to Dietrich Kappe are not dependent on any single engine, but the Stein network uses some Leela data.

[Daniel Shawul]

Using, for instance, both SF data and Nodchip SF learner (thus qsearch in learning phase) might be considered too much of a "copy".
Using, own data and SF learner might be considered borderline.
Using, own data, and own learner, might be OK if NNUE technology (and subsequent copy-pasting) is indeed view as a "lib".

There is a fourth option, namely using SF data and own trainer. This is the approach that Allie currently uses when it uses the Stein network in TCEC. The trainers for Leela nets according to Dietrich Kappe are not dependent on any single engine, but the Stein network uses some Leela data.

What you are proposing here i.e. to apply same rules for NN engines that they don't use Lc0 data, as you plan to do for NNUE engines that they don't use SF data, applies equally well to alpha-beta engines as well. Infact, it would probably exclude i don't know 70% of the serious AB engines is my guess.

Some examples:
a) The popular Zurichess quiet-labelled.epd with 750k is scored with Stockfish-7.
I would be surpized if an engine developer hasn't tried that at some point.
b) Alvaro Begue has 3 millions fast SF 8 games for tuning and training neural networks.
c) Chris just published 41 million fens scored with SF11 for tuning
d) Ed had PGN games and epds scored with Stockfish in the past
It makes sense people score positions with the best available chess engine at the time.

When you use CCRL data, you use games and positions scored by different engines, and if you choose >3000 elo you choose the strongest engines as teachers. If you use human data only, you get weaker results, even CCRL gives weak nets usually. With supervised learning, you need a teacher basically (so cloning by your definitions), be it stockfish/lc0, humans, CCRL etc Reinforcement learning, that you learn from your self became popular only recently. And it is a very slow and costly process. So is your proposal to remove all engines tuned/trained with supervised learning (those who had a teacher for training)?

So I ask again what is Ok for you, and how do you plan to apply your rules to alpha-beta engines as well ?

Daniel

[Raphexon]

+Allie's net is trained on T60 games (among others) and before some recent updates Allie's eval almost always mimicked Leela's eval. (Just inflated massively)

If you replace SF's multithreading with ABDADA (or YBWC) and train a net a little differently you quickly have something that's just as *unique* as Allie.

A0lite ‘mimics’ leela’s eval because they both ‘mimic’ A0’s eval (read the papers). You’ve correctly identified that there is a parent engine, you’ve just identified the wrong one (and if you claim that a0lite’s 95 line python implementation of mcts/uct is a clone of leela, I have a bridge to sell you).

I was talking Allie(+Stein), not your A0lite.



I know you don't use the same training data, you didn't use the same teacher engine, your code is completely different.

[Alayan]

When the eval architecture is actually different, the source of the training/tuning data is of little importance. The zurichess set used to tune classical evaluations of non-SF engines never was an issue.

When you copy the eval architecture, then where the training/tuning data comes from becomes a worry because that's the last barrier to having basically the same eval (the Stein net were better performers than the main leela nets thanks to some SL optimizations, but they behaved very much the same).

Realistically, any NNUE engine authors could take some old sergio net, put it through a short SL phase that doesn't change much its strength but change all the floats making up the weights so the source can't be traced, claim it as his own achievement, and get a net that's a few elo short of SF's one. This is trivial, and simex has proven to be unable to detect this kind of similarity anyway.

[Madeleine Birchfield]

When the eval architecture is actually different, the source of the training/tuning data is of little importance. The zurichess set used to tune classical evaluations of non-SF engines never was an issue.

When you copy the eval architecture, then where the training/tuning data comes from becomes a worry because that's the last barrier to having basically the same eval (the Stein net were better performers than the main leela nets thanks to some SL optimizations, but they behaved very much the same).

Realistically, any NNUE engine authors could take some old sergio net, put it through a short SL phase that doesn't change much its strength but change all the floats making up the weights so the source can't be traced, claim it as his own achievement, and get a net that's a few elo short of SF's one. This is trivial, and simex has proven to be unable to detect this kind of similarity anyway.

We could do the same as for engine uniqueness and try to find definitions for neural network uniqueness. When is a neural network a clone or a derivative of another neural network, and how much reinforcement training is needed to make a neural network different enough that it could be considered unique?

[Daniel Shawul]

When the eval architecture is actually different, the source of the training/tuning data is of little importance. The zurichess set used to tune classical evaluations of non-SF engines never was an issue.

You get judged by the same standards you set. Any engine trained with supervized learning is a "clone of its teacher" period by your definition.
"It was never an issue" is an excuse. It does become an issue _especially_ when you apply one standard to NNs and another to AB engines.

When you copy the eval architecture, then where the training/tuning data comes from becomes a worry because that's the last barrier to having basically the same eval (the Stein net were better performers than the main leela nets thanks to some SL optimizations, but they behaved very much the same).

The days of writing hand-crafted evaluations (selecting features manually) is over. If that is the "art" you are missing as Andrew said, then tough luck.
I told Larry Kaufman the same thing in the early days A0 announcment. For image recognition, there are a couple of successful architectures like ResNet, DenseNet etc. that every one uses without care for "originality". So if you want to divide up the net architectures to one NN engine per architecture, you won't have many. This is all ridiclous ofcourse.

Realistically, any NNUE engine authors could take some old sergio net, put it through a short SL phase that doesn't change much its strength but change all the floats making up the weights so the source can't be traced, claim it as his own achievement, and get a net that's a few elo short of SF's one. This is trivial, and simex has proven to be unable to detect this kind of similarity anyway.

NNUE has one simple architecure (3-layer dense net) and one trick (incremental evaluation with king-piece association inputs). You can take advantage of that now without giving a damn about what anyone says and hopefully improve upon it in the future, thus contributing. Do you want them to drop the incremntal part of it, or the "king-piece" association part of it for you to deem then original ? New technologies motivate engine authors to work on their engines, especially now that you can get close to the best with NNUE's help...
If you are like Andrew who on the one hand wants to get back to DivP by all means, and on the other is confilicted because he set a standard so high by hist past rumblings against NN, you will end up tip-toeing to what your heart wants (the NN).

[AndrewGrant]

When the eval architecture is actually different, the source of the training/tuning data is of little importance. The zurichess set used to tune classical evaluations of non-SF engines never was an issue.

You get judged by the same standards you set. Any engine trained with supervized learning is a "clone of its teacher" period by your definition.
"It was never an issue" is an excuse. It does become an issue _especially_ when you apply one standard to NNs and another to AB engines.

Well then you'll be happy to know that Ethereal is not trained in any way shape or form on Stockfish.
No Stockfish games, No Stockfish searches, No Stockfish PVs, No Stockfish Qsearchs, No Stockfish Evaluations.