My failed attempt to change TCEC NN clone rules

[Gian-Carlo Pascutto]

the poster child of boom bust AI investment development cycle. Each boom has been accompanied propaganda-wise on dreams of “well if they can beat humans in chess ...” and each bust on failure to actually deliver on the dreams. So, I would posit, computer chess is about bullshit also. Extracting copious investment sums from greedy capitalists, isn’t that what AZ is really about? Sure, Go now as well. And one of the reasons there are a few sleazy crooks hovering around the upper echelons of it.

<cue DeepMind posting horrendous financials>

second, at programmer level, it’s about ego and identity. You can tell by the huge importance given to the name of the thing. The objection to Alliestein (or any other equivalent example) is that the original has been renamed, an act of stripping identity away from the actual originators. If it were called LC0WithSomeChangedBits the whole thing would be different. But Alliestein and something in the notes about thanks to LC0 is not the same thing at all. Alliestein says “I own this, its part of me now, my ego/identity is in it”, and “speak to me differently now, I am somebody, I created a chess engine”.
Likewise the people/person recognised for hard work in LC0, see this seizure of status (equivalent status to them) as kind of cheat “hey you can’t do that, I am in that engine, you are erasing me by the renaming and publicity act”.
See? It’s all about the naming. People in open source projects object when the project is renamed as some kind of new entity, and some new person gets a cheap route to status, which is really belonging to the originators, on the originators back and hard work.
Same with DeusX. The renaming and attempts to hide origins. That case more obviously about ego/identity because of the accompanying displays of narcissism.

I think you are right.

PS. "Leela Chess Zero" is a much nicer name than lc0 don't you agree.

[chrisw]

the poster child of boom bust AI investment development cycle. Each boom has been accompanied propaganda-wise on dreams of “well if they can beat humans in chess ...” and each bust on failure to actually deliver on the dreams. So, I would posit, computer chess is about bullshit also. Extracting copious investment sums from greedy capitalists, isn’t that what AZ is really about? Sure, Go now as well. And one of the reasons there are a few sleazy crooks hovering around the upper echelons of it.

<cue DeepMind posting horrendous financials>

second, at programmer level, it’s about ego and identity. You can tell by the huge importance given to the name of the thing. The objection to Alliestein (or any other equivalent example) is that the original has been renamed, an act of stripping identity away from the actual originators. If it were called LC0WithSomeChangedBits the whole thing would be different. But Alliestein and something in the notes about thanks to LC0 is not the same thing at all. Alliestein says “I own this, its part of me now, my ego/identity is in it”, and “speak to me differently now, I am somebody, I created a chess engine”.
Likewise the people/person recognised for hard work in LC0, see this seizure of status (equivalent status to them) as kind of cheat “hey you can’t do that, I am in that engine, you are erasing me by the renaming and publicity act”.
See? It’s all about the naming. People in open source projects object when the project is renamed as some kind of new entity, and some new person gets a cheap route to status, which is really belonging to the originators, on the originators back and hard work.
Same with DeusX. The renaming and attempts to hide origins. That case more obviously about ego/identity because of the accompanying displays of narcissism.

I think you are right.

PS. "Leela Chess Zero" is a much nicer name than lc0 don't you agree.

haha! yes, of course, but is clear the renaming/reidentification of the series has each time respectfully acknowledged the antecedents. Alpha Chess Zero to Leela Chess Zero to LC0. DeusX and Alliestein and others, I don’t really keep up with this stuff, break that respect and effectively try to own the entity as new. Which is the root of the problem. Identity theft. They are stealing what it is that you felt was you. Your identity in the machine. No manner of acknowledgements in the notes, or at the end of posts, can compensate, unless and until the engine itself has diverged so far from the original as the “you” is no longer in it, but that’s a long way off from here.
Meanwhile, all the arguing about which bits are more important, ie diverting everything to the technical, is proxy arguing, because nobody wants to admit, or are perhaps too immature to see, the psychology involved. The real criminals, over and above the identity theft, are TCEC, for they provide the arena, or Colosseum maybe, where the hurts are inflicted, for what you accurately describe as “entertainment”.

[MikeB]

You certainly have made very strong argument that Allie is a clone , regardless of what anyone points to as being different. And NN (scorpioNN, Stoofvlees) or MCTS (KomodoMCTS) are not clones.

Not sure if it will change anything under the current infatuation state with anything neural.

If you run SIMEX on neural network engines (other than Stoofvlees), you get similarity of over 60%. That’s because they all derive from the alpha zero pseudo code which dictates certain neural network structures. They’re all “conceptual” clones of alpha zero, not lc0, which is itself a clone (though here clone is not a really useful term). Even a from scratch 100 line python implementation of mcts will produce the the same moves on the same net once multithreading is disabled. It’s in the nature of deterministic algorithms.

If you find this sort of hand waving argument from pictures compelling, I have a bridge to sell you.

You do great work with your nets , not sure why you are getting so worked up over this. Appears to be an overreaction to back of an envelope comment to back of the envelope type post. 'Simmer down now' ;>0 -> just an old saying from a sitcom many years ago meant to make you smile. You can keep the bridge , but I will take your NNets any day ;>) Peace my friend.

[smatovic]

If you agree with the above, then you need to have something in place to make sure engines are "different". It's not as big a deal when you just have a couple duplicates like with Lc0 and DeusX, and I don't have an issue with different groups that are trying different approaches, but when you can take GPL code and 20 people can churn out 20 flavors of the same engine, you need something in place to limit them, or you end up with just a ranking tournament of clones.

Allie is not a clone of Lc0. By far the most important part of code that Allie uses from Lc0 project is the cudann backend that Ankan wrote. Here is a direct quote from Ankan when I asked him if he had any concerns about Allie using this code and playing in TCEC:

"Hi. No, i don't have any issues. In fact I am happy that my code is used in multiple engines

Btw, congratulations on great performance by Allie at TCEC"

That code is great, very important and all praise to Ankan for writing it, but that code all by itself is a far cry from a chess engine. So you see Allie is not a clone or a copy of Lc0. And other developers are encouraged to use that cudann backend all they want to make actual NN chess engines. Feel free and enjoy.

That really should put this issue to bed IMO.

Maybe I am the only one, but I find it some kind of misleading to state that a) Allie is original code and b) you copied parts of LC0.

https://github.com/manyoso/allie

What is Allie?

Allie is a new and original chess engine heavily inspired by the seminal AlphaZero paper and the Lc0 project.

How is she related to Leela?

Like Leela, Allie is based off of the same concepts and algorithms that were introduced by Deepmind in the AlphaZero paper(s), but her code is original and contains an alternative implementation of those ideas. You can think of Allie as a young cousin of Leela that utilizes the same networks produced by the Lc0 project.

is imo conflicting to:

What bits are used from the Lc0 project?

Here is what Allie uses from the Lc0 codebase:

Protocol buffers for an NN weights file
Code for discovering/loading the NN weights file
Backend code for GPU to get evaluations given an NN weights file

--
Srdja

[Ovyron]

Stoofvlees exists because at some point the CSVN thought it was OK to allow two engines from the same author (and ignored my objections over this). I entered two engines, and Stoofvlees nicely gave a full point to Deep Sjeng when they played each other.

Whether that was done fairly I'll leave in the middle, as part of the point is that you can't tell.

I think this is a matter that can't be decided by a rule that "covers everything", because all cases are unique, a rule might be best in one and worse in another. So I agree with crem that the rules should remain loose and versatile, and allow intelligent humans to make the best decisions in edge cases.

For instance, when it was found that Fritz 15 was nothing more than some Rybka 5, Vasik Rajlich wasn't allowed to keep both in TCEC:

One more new participant enters TCEC. This is one of the top 10 engines that was missing last season from the event – Fritz 16. It will start in Third Division, taking the place of Rybka and officially retiring it from the roster.

Which makes sense, lest we allow people to have multiple engines if they change their name between versions. However, other rating lists and competitions decided to keep both Fritz and Rybka as if they were unique, and continue to test Rybka 4.1 to this day even though they stopped testing all other versions of Rybka, so with Fritz being its new name they should have made the replacement.

I think that we should then stop looking at an engine's author, so if an author creates 6 different engines from scratch and all of them are different, we should treat them as different entities and allow them all. Conversely, if 2 authors create two independent engines from scratch but they end up playing the same, we have no reason to have them both in.

If Allie only plays 60% similar to what it was supposedly cloned off, I don't know what all the fuss is about. Seriously, when I read "NN clone" in the title, I expected some +90% move matching, so it should be fine, just, also allow Stockfish derivatives that show similarity with it and others of around 60%, because if the engine is strong enough that it can play weak on purpose to avoid playing similarly, then engine authors that begin from scratch are wasting their time re-inventing the wheel, and the answer is just creating an engine "top-bottom", something unique in the moves it plays, and not in how it was constructed.

[chrisw]

Stoofvlees exists because at some point the CSVN thought it was OK to allow two engines from the same author (and ignored my objections over this). I entered two engines, and Stoofvlees nicely gave a full point to Deep Sjeng when they played each other.

Whether that was done fairly I'll leave in the middle, as part of the point is that you can't tell.

I think this is a matter that can't be decided by a rule that "covers everything", because all cases are unique, a rule might be best in one and worse in another. So I agree with crem that the rules should remain loose and versatile, and allow intelligent humans to make the best decisions in edge cases.

For instance, when it was found that Fritz 15 was nothing more than some Rybka 5, Vasik Rajlich wasn't allowed to keep both in TCEC:

One more new participant enters TCEC. This is one of the top 10 engines that was missing last season from the event – Fritz 16. It will start in Third Division, taking the place of Rybka and officially retiring it from the roster.

Which makes sense, lest we allow people to have multiple engines if they change their name between versions. However, other rating lists and competitions decided to keep both Fritz and Rybka as if they were unique, and continue to test Rybka 4.1 to this day even though they stopped testing all other versions of Rybka, so with Fritz being its new name they should have made the replacement.

I think that we should then stop looking at an engine's author, so if an author creates 6 different engines from scratch and all of them are different, we should treat them as different entities and allow them all. Conversely, if 2 authors create two independent engines from scratch but they end up playing the same, we have no reason to have them both in.

If Allie only plays 60% similar to what it was supposedly cloned off, I don't know what all the fuss is about. Seriously, when I read "NN clone" in the title, I expected some +90% move matching, so it should be fine, just, also allow Stockfish derivatives that show similarity with it and others of around 60%, because if the engine is strong enough that it can play weak on purpose to avoid playing similarly, then engine authors that begin from scratch are wasting their time re-inventing the wheel, and the answer is just creating an engine "top-bottom", something unique in the moves it plays, and not in how it was constructed.

If the naming was correct, truth would follow. If the naming is incorrect truth does not follow. Incorrect naming creates Fake News. For example, some kid watches TCEC and posts some wild title to a thread “Alliestein smashes Leela at TCEC!!!! End of an era!!!!” This would be Fake News, right? Whether the program wins 51.5 to 51.0 or not. It happens because the kid is misled by the naming. Stop calling it Alliestein and start calling it LC0-Alliestein variant or the other one LC0-Deusx variant or LC0-Fat Fritz variant and everything will be correct. The “author” should make it clear himself in the name. The “author” won’t because the author has some degree of narcissistic personality disorder and wants all the glory reflected back for himself. Well it’s a straight cheat. Name of original base first, name of variant second, joined together. Or as Confucius wrote:

Confucius. The Analects states that social disorder often stems from failure to call things by their proper names, that is, to perceive, understand, and deal with reality. His solution to this was the "rectification of names". ... If names be not correct, language is not in accordance with the truth of things.

[Ovyron]

It happens because the kid is misled by the naming.

So is this kid someone that reads a title and a post and believes everything that she reads without critical thinking? I think this kid should learn to research a subject first before believing everything she reads, otherwise, I don't care about the fake news that this kid believes.

And, anyway, if LC0 is a clone of AlphaZero, are you suggesting that they should rename their engine to AlphaZero-Leela? Should all A/B engines start their names with the name of the first engine that used Alpha-beta?

When does it stop? Because, I don't see "Turing" at the front of every computer ever named, Stockfish doesn't even have "Glaurung" in the name even though the first version was mostly Glaurung.

If everything was solved by keeping everything the same but just naming some X as Y, then the starting problem seems nitpicky. I don't think the name Allie has is significant.

[chrisw]

It happens because the kid is misled by the naming.

So is this kid someone that reads a title and a post and believes everything that she reads without critical thinking? I think this kid should learn to research a subject first before believing everything she reads, otherwise, I don't care about the fake news that this kid believes.

The kid comes from a space where there are thousands of kids, and it is not senseful to solve the problem by addressing the kid. It is possible by addressing the root.

And, anyway, if LC0 is a clone of AlphaZero, are you suggesting that they should rename their engine to AlphaZero-Leela?

Leela Chess Zero recognised Deepmind’s origination without transgressing the probably legally protected name Alpha Chess Zero. Good choice of name.

LC0 recognised Leela Chess Zero and maintained the link back to AZ Zero in so doing. Good choice of name.

Should all A/B engines start their names with the name of the first engine that used Alpha-beta?

Silly question.

When does it stop? Because, I don't see "Turing" at the front of every computer ever named,

Stockfish doesn't even have "Glaurung" in the name even though the first version was mostly Glaurung.

Not a relevent comment. Stockfish authors included the Glaurung author.

If everything was solved by keeping everything the same but just naming some X as Y, then the starting problem seems nitpicky.

If you create a straw man by argumentum ad absurbam amongst other fallacies, then anything can be made to be anything you want. But it isn’t what is known as an argument.

I don't think the name Allie has is significant.

But you didn’t back up what you think with anything that could be called a rational argument. I don’t see, I don’t care and I don’t think are not arguments.

The point of Confucius still stands, several thousand years on. It’s the name that’s the problem.

[mhull]

So for the record, you agree that the lc0 cuDNN/CUDA backend is a very critical part to the strength of an NN-engine?

I think I've been very clear and said *numerous times* that the cudann backend is a very important critical piece of code...But to be very clear, the sentiment I was relating above was the unsolicited opinion of a top lc0 developer.

I'll explain you why I highlight this:

That code is great, very important and all praise to Ankan for writing it, but that code all by itself is a far cry from a chess engine. So you see Allie is not a clone or a copy of Lc0.

See, the thing is nobody has a good definition of "clone". It was pointed out in the beginning, and if anything, it looks like this thread was an attempt to define the term for NN engines. Good luck! I don't think anyone agreed on one for AB engines and those are quite a bit older and more understood.

I think I understand somewhat where crem's frustration comes from. He thinks computer chess is a programming competition. It's logical, because it superficially looks that way. But nothing could be further from the truth. From a competition point of view it seems most sensible to look at where an engine gets its strength from. Hence, it looks senseless from this POV that lc0 is losing to competitors that use lc0's code and get a lot of strength from it. (This is even more painful because the results are somewhat random and lc0 could easily lose to an engine using its code that is actually a bit weaker! It is literally possible to take lc0, make it a bit weaker, and still hope to win TCEC/CCC/whatever)

But computer chess is not a programming competition. Computer chess is entertainment. If it were a competition, there'd be no Komodo and KomodoMCTS in TCEC - and I'll leave it at those two most obvious examples to not open another dozen cans of worms. From an audience point of view it is more interesting to have lots of strong engines, preferably that still differ enough in how they see a position so there's enough decisive games. The audience doesn't give a f--- about the graphs posted on the first page. They care - at most a little - about engines that are so close that it's like watching a self-play game. Details like how the NN backend was written - even if it took months of blood, sweat, tears and access to non-public GPU programming details - are not relevant, as long as the engine plays a bit differently and the author can explain a bit what he did. So hoping that the rules address this is futile - there's a fundamental conflict of interest. There's quite a bit more people watching and discussing TCEC than there are watching the ICGA Computer Chess World Championship, right?

It took me quite a while to come to that realization when I was originally involved in computer chess, and after I did, I quit.

(Don't ask me what I'm doing here again now, it's a long story involving ataxx engines...)

In particular, computer chess has become about human interest in computer chess and not really about chess. The reason I say this is that human beings, who cannot play half as good a game of chess as the computers/programs in the competition, are dictating to these far superior players from which start positions they will begin their contests. What GM would permit himself to be dictated-to and/or hobbled in such a manner? None. But relative wood-pushers feel qualified to select openings that these super-players will play. There's something fundamentally wrong with that, IMHO.

I was particularly disappointed to see Lc0 constricted in this way because in training, it seemed to learn fundamentals (and even advanced concepts) about openings without a book. So why is it forced to play openings it would likely never choose? The reason I've heard given is that because it would be boring to watch it play the same openings (like winning ones), assuming it would play that way. This is perhaps why the World Championship between Alekhine and Capablanca is not mentioned very often, since it was just a bunch of Queens Gambit Declined games. Boring. So let's all make sure our Computer Chess matches aren't boring for humans! If Lc0 should for all intents and purposes, "solve chess" out of the opening or otherwise discover on its own which opening is best for winning chances, competitions like TCEC will NEVER discover it.

So what's TCEC really about? It's certainly not about the game of chess since none of the participants are allowed to play the opening on their own recognizance. And if they were allowed to do so, maybe Lc0 will sweep the field with it's one boring opining to win them all? Maybe? We'll never know, will we.

[carldaman]

I like to run engine matches where no book is used, since I'm interested in how an engine conducts the opening on its own, but I do also like short opening books for the sake of variety. I believe it's important to see how engines handle various openings because we are naturally interested in many different opening lines as humans. How useful can an engine be for analysis if it has no clue in most openings or resulting middlegames?

Ironically, this variety may yield a more accurate rating for most engines, since, despite their strength, the engines don't always know what's good for them in terms of openings. They play what they assume is best, but that is automatically flawed because the evaluation function is not perfect and the necessary depths to more deeply assess what's best are out of reach when typical time controls are used. Chess isn't solved and the opening remains the weakest phase of the game for even the strongest engines, as a consequence of this.
It often turns out that many engines may perform better in openings that they would (or could) not enter on their own.

Now, LC0 is an exception to this weakness, due to the new NN paradigm, where it was trained to learn the game from scratch, and always from the familiar starting position. Its strength is great in the openings it plays on its own. However, it remains to be seen if the way LC0 is trained is the most optimal. We are already seeing other approaches where the training is getting away from the 'zero' method, incorporating the games of actual GMs or even those of Stockfish.

[As an aside, LC0 also has some tactical shortcomings, which means we can't always trust it, even in openings it favors, and probably so much less in those it does not. This is a serious flaw, IMO, but one that could be overcome, perhaps as its code and training methods are refined any further. Furthermore, one could bring up its sub-optimal play in endgames, or positions where it's down a piece (as in Larry Kaufman's odds matches).]