Strange Lc0 TCEC performance

[jkiliani]

Watch a game of TCEC -- Leela best moves will match the opponents predictions for large chunks of the game. This "humanlike" marketing gimmick will die at some point.

Indeed, I have seen that the move predictions match in many cases, especially when playing stronger opponents. But they do differ in a minority of moves, and those moves determine the direction where games end up positionally. About "humanlike": I did not say that. Leela is humanlike only to the extent that she is vulnerable to tactics, more than AB engines but still less than humans. The reason people like Leela games is not that she plays like a human, but that she regularly comes up with positional sacrifices that human players would never dare, yet end up working. Her board evaluation is sufficiently different from that of AB engines that many NN-AB games end up with large discrepancies in eval, where sharp play and decisive games develop until one of the engines is proven wrong. Obviously, there have been lots of boring Leela games as well, but on average, a Leela-AB game has a higher chance to be interesting than a game between AB engines (even if it's a case of a Leela delusion leading to a loss).

[George Tsavdaris]

Watch a game of TCEC -- Leela best moves will match the opponents predictions for large chunks of the game.

This is clearly misleading since this happens with all pairs of strong opponents.
And yet after only 6 games in the bonus games between Leela 10520 and 10800 against Ethereal 10.81 with 6 draws out of 6 games where Leela in all controlled the game, in 2 it completely outplayed Ethereal in the middlegame.
So how to say Ethereal matched Leela's predictions when it got outplayed?

Leela 10520 - Ethereal 10.81



Ethereal 10.81 - Leela 10809


And this was full 43 cores TCEC Ethereal, while Leela was struggling playing with 1 GPU due to the thermal problems!
8000$ versus 900$ hardware in favor of Ethereal.

In the aforementioned 2 games i said, Leela didn't won since she couldn't find the winning moves in the endgame:
In the one game the 49.Rd7! was winning much easier instead of 49.R2d5, but still this was also winning and Leela missed 54.Bf5 that would win and played 54.Kd5 with the Kc6? plan that draws.
And in the other game 48....g5+ would win instead of Ra1?!


The games (they contain also the 3 Leela - SF8 43-cores bonus games that were also all draws)

[frankp]

My perception from these games and others is that leela is not good (yet) at assessing the drawishness of early endgame positions, or it would probably score much more highly by converting its middle game advantages into a won ending. Not talking about late endgame tb positions (although I guess tb would help even in the late middlegame) but classic many pawns and rook or bishops - where the configuration of pawn etc is often key.

I guess it is just a question of training.

[chrisw]

Even better, if the Chess.com tournament is a success, you'll be left alone by NN fans at TCEC, wouldn't that be great?

I hope and pray.

Not on a CPU you won't

On the topic of playstyle -- I'm yet to see anyone which actual chess knowledge show that Leela games are different in any capacity.

It is not really possible to make "chess knowledge" statements about NN. Because. Insert essay here.

It is possible to point at certain moves and see what is generally being evaluated. Quiet often. Different to AB engines (usually).

[Laskos]

Again spitting gibberish with some 5% here and there, while missing the general picture? Nvidia 11 series is just 2-3 months from now, and as we know, they will come with some massive improvements. And by the way, my OC i7 4790 is at least on par with say stock i7 7700. All in all, you nitpick (with misplaced wording) on the general picture that on fairly balanced CPU/GPU home hardware, Lc0 with a good net is on par with SF8 on 1-2 threads in blitz. And probably soon will be on par with SF8 on full CPU, and one could casually say "Lc0 is SF8 level". Generally, behave, and don't use misplaced language.

Sorry but this childish fanboy behaviour of yours, patronising and telling ppl what to do is not gonna work.
You simply ignore or disregard facts and offer no arguments.
The only true claim of yours is that Lc0 on 1060 is on par or maybe slightly better (depending on actual TC) than SF8 on a single strong core in blitz. Everything else is, either gross exaggeration or just invented stuff.
You talk about massive GPU improvements that are gonna be at best 15% and ignore the fact that moneywise your 1060 is on par only to 8 core/16 threads Ryzen that gives 2.5x more nps than your 5 years old CPU, and powerwise 1060 is on the level of 16 core/32 threads Threadripper.
To me it seems you are just acting angry coz your precious CPU (with only ancient 4 cores) is much, much worse than some much cheaper AMDs.

And you talk about some improvement in Lc0 that would magically enable it to go from on par on single core to on par with 4 cores while (and I tested that thoroughly) the glorified 10780 net has 5% worse performance than 10445 vs SFdev in blitz, i.e. there is clear regression and 20x256 nets have saturated. Again clear case of fanboyism on your side.
Are you even aware that strongest Lc0 on 1060 can't beat Droidfish on 3 years old Samsung Galaxy S6?
Hey, Lc0 on state-of-the-art middle range GPU is not better than old SF (SF9) on old cellphone!!! So much about equal hardware and other bullshit.

Lol, are you denying the fact the testserver nets are improving dramatically on mid-range spans? How many blitz games have you played? Smartphone games for that matter? A normal reasoning would be that if on my balanced hardware, testserver nets went pretty quickly from Shredder 13 level on one thread to SF8 level on 2 threads in blitz, then, as LR window can still be lowered several times to get to the global optimum, it is fairly likely it will get to the full CPU SF8 level. Be it that AMD Ryzen 7, or a factor of 1.7 in effective speed compared to my OC i7 bothers you in this progress? 50 Elo points at best? When you want to impress people with your "knowledge", you make intentional or stupidity confusions about linear factors and logarithms of those, and you will go at length explaining how a 32 core CPU will smash anything. Wait 5-6 months, and we will see SF8 on uber-duper logarithmic in cores Elo-wise 32 cores being smashed by Lc0 on my puny single 1060. Your big mouth, which is actually not that stupid, it is moderately stupid, will invent some more, maybe you will come with some FPGA solutions for regular engines, as that Andrew Grant seems to imply GPUs are very specialized hardware. Lol.

[chrisw]

Without crippled hardware, Leela would be on first place.
Deux maybe second place!

99% will go offline without NN engines at TCEC.

Shame to see people leave, but will return the chat to actual talk of chess development and play.

I would hope NN-fans would be happy that TCEC did all this work to get their own server.

It's pretty clear by now that actual chess engine development in the next few years is going to be about how to use neural networks most effectively, since Leela has demonstrated the AlphaZero concept rather conclusively. Alpha-Beta searchers may have a clear edge with tactics and the use of tablebases, but they can't compete with a strong NN engine in quiet positional play. In addition to improvements on the tree search and neural net architecture, the next few years will see people trying to fuse NN and AB engines to combine the strengths of traditional engines (tactics, endgame when using TB) with the strengths of neural networks (opening, quiet midgame). As AB-engines go, Ethereal is a very fine piece of engineering, but a successful fusion of its search with Leela's would make it world-class.

There's a reason NN has so many fans, and it's not (mainly) the playing strength: It has revived computer chess by producing actually exciting games that break with preconceptions about how you're supposed to play, and yet manage to win somehow. Chess.com has realised this, which is why they are going all in for GPU infrastructure. I'll give TCEC points for effort, but they'll have to up their game in Season 14 to get viewers back.

Clear to who? Until you can write an NN that runs as well as AB on a CPU, then the only thing you are proving is that massive specialized hardware trumps a general purpose processor. Thats it. Cut and dry. No further interest.

Merging engines is something people with no clue what they are talking about say. Its condescending to even suggest it to an author.

No offense to Erik or to chess.com, but they have not realized jack sh*t. They are playing with absurd settings, no opening book, trying to force hyper threading with pondering. They even made their release without telling authors before hand. They don't have the slightest clue about ANYTHING that goes on in this forum. They have $$$, and they see NN fans, and business does the rest.

Danial Shawul has done a good bit of work on "merging engines", but that's possibly not what you meant. It would not surprise me if AB paradigm held against AZ. Those NNs are good at non-linearity, while traditionally the AB polynomial eval isn't, some people have tried to get wilder terms in, but in they end, it all was added up, and there's only so much one can get away with on individual terms. But.

There's no reason why forcedly time-cheap CPU evaluators can't train better on chess game data. Nobody said chess has to broken up into bits and then all added back together again as a Law of AB. Chunks of processed "Knowledge" can be trained on with a degree of non-linearity using game databases, and that non-linear eval used in a CPU program without the time-cost of an NN. Point 1 in the defence of AB paradigm.

Point 2. AB programs are already exceptionally proficient at move sorting (if one measures how often a search beta-cut is actually the first selected move, I get 99% success rate on the occasions I tested for that). Sometimes they get it wrong, but then NN Policy gets it wrong too.

Pruning decisions in AB remain more or less in the dark ages. I'm astonished at how much, and with so little chess sense, Stockfish prunes, but it works, which is what counts. So, with smartened up pruning/extension decisions and smart non-linearity in evaluation, and AB could well stay on top. My opinion.

[frankp]

AB engines are very successful mimics of the 'human' approach to chess, hard coded into their program. It would make sense for them to incorporate the AI approach if this becomes superior - positional evaluation etc.

I am still at the fascination stage with leela, where you can give a program the rules of chess, a NN structure and let it learn for itself from self-play to become a very strong chess entity in a relatively short time - without humans having to tell it what to do at every stage based on decades of research into search techniques and encoding positional evaluation.

(Hard to understand the antipathy towards the project, particularly on this forum.).

[chrisw]

AB engines are very successful mimics of the 'human' approach to chess, hard coded into their program. It would make sense for them to incorporate the AI approach if this becomes superior - positional evaluation etc.

I am still at the fascination stage with leela, where you can give a program the rules of chess, a NN structure and let it learn for itself from self-play to become a very strong chess entity in a relatively short time - without humans having to tell it what to do at every stage based on decades of research into search techniques and encoding positional evaluation.

(Hard to understand the antipathy towards the project, particularly on this forum.).

Please quote, from one smart person on this forum, the raw data that says there is "antipathy to the project". Else Strawman Fallacy.

[frankp]

AB engines are very successful mimics of the 'human' approach to chess, hard coded into their program. It would make sense for them to incorporate the AI approach if this becomes superior - positional evaluation etc.

I am still at the fascination stage with leela, where you can give a program the rules of chess, a NN structure and let it learn for itself from self-play to become a very strong chess entity in a relatively short time - without humans having to tell it what to do at every stage based on decades of research into search techniques and encoding positional evaluation.

(Hard to understand the antipathy towards the project, particularly on this forum.).

Please quote, from one smart person on this forum, the raw data that says there is "antipathy to the project". Else Strawman Fallacy.

This is irony - yes? Or do you expect me to define who and who is not smart on this forum?
Whichever, my interpretation on some comments seen here - right or wrong as it may be. (And of course comments elsewhere.). And btw, not you. YMMV.

[chrisw]

AB engines are very successful mimics of the 'human' approach to chess, hard coded into their program. It would make sense for them to incorporate the AI approach if this becomes superior - positional evaluation etc.

I am still at the fascination stage with leela, where you can give a program the rules of chess, a NN structure and let it learn for itself from self-play to become a very strong chess entity in a relatively short time - without humans having to tell it what to do at every stage based on decades of research into search techniques and encoding positional evaluation.

(Hard to understand the antipathy towards the project, particularly on this forum.).

Please quote, from one smart person on this forum, the raw data that says there is "antipathy to the project". Else Strawman Fallacy.

This is irony - yes? Or do you expect me to define who and who is not smart on this forum?
Whichever, my interpretation on some comments seen here - right or wrong as it may be. (And of course comments elsewhere.). And btw, not you. YMMV.

By smart, I mean excluding anyone commenting "Leela yah boo sucks", for example. I don't think those exist here either.

There isn't antipathy towards the project. One quote would be enough. Do you have one to show us?