Leela Chess 1/2 would be stronger than LC0

[Albert Silver]

The Lella Chess project is trying to "replicate" the Alpha Zero project.
The Alpha Zero project is based on very different concepts from classical AB engines and they did not use AB engines to train the networks.
Remember the A0 team isn't stupid nor ignorant, they are pretty smart.
The A0 team proved (at least to me) that their methods really works, you just need to implement them very well.
The LC0 project is also proving these methods works. The main problem of LC0 is in my opinion the hardware. The Google team has access to fabulous hardware while the LC0 team don't, so they will have to wait quite a bit longer to train the networks. Unfortunately also during game play it is very limited in nodes per second, maybe in the future someone will aqquire some of those TPUS and test LC0 on that hardware. While you can run a classical AB engine on your PC and get very powerfull play from it you won't be able to get high performance from LC0 on a "normal" PC, at least on mine with a Radeon 4600 HD and a core i7 4 core Nehalem I won't get anywhere with LC0.

The problem is the following. AG0 like AG is based on MCTS which was the only successful concept in Go. So there was nothing new (at least in AG case) except monstrous hardware. With AG0 they introduced reinforcement learning. This is also fine, because their game model is Go where we have known for quite a while (well before Google got its hands on it) that MCTS+NN approach works. Additionally to reinforcement learning, in AG0 they unified policy and value network in one as another novelty. And we can say this has been reasonably demonstrated that it works for Go.
Chess is a different game altogether and just blindly assuming the concepts from Go would work equally well without proof is plain wrong.
Some take that lousy preprint as a proof, some are intimidated by Google and would take whatever they say at face value, fine, but I disagree with that.
We don't know if MCTS (or UCT in what ever version) would work for chess (there are strong indications it doesn't). We don't know if combining policy and value network in one NN would work for chess (so far judging by atrocious LC0 tactical performance it doesn't).
Yes LC0 is based on L0 simply because A0 is based on AG0. And what we know so far (beside awful tactical performance and very strong positional performance of LC0):
1) Training is saturated (it is a current state no matter how much some ppl bury their head in send). Whether it is because of bug in training code, small network size, bad training process, or simply too shallow training games due to the lack of monstrous hardware that Google had at their disposal is to be seen.
2) Despite having relatively large net (half of the A0) performance based on extrapolation to equal hardware as A0 (4TPUs) is 300-400Elo below A0. And we know that increasing net size brings initially very little if anything because of much slower evaluation. Doubling net size usually slows down nps for factor 2.5-3. When going from 6x64 to 10x128 Lc0 gained a lot. When going from 10x128 to 15x192 much, much less, and I'm talking here about testing with fixed number of playouts.
3) Scaling performance is not nearly as good as some expected it would be (ofc I would say) and is totally different to what has been presented in that lousy preprint. It is 50-60 Elo per doubling of playouts.

Myeah... you know that the ratings posted for each NN are based on self-play testing, right? And from the day it was switched to 15x192 on April 30, it advanced 140 Elo by May 6. Granted self-play exacerbates progress, so this will not translate to 140 Elo for play against other engines, but progress it is. I'll gladly test two NNs, say those two I just cited, and run a match against an engine, Maybe Baron, DIsaster Area, or some other 2900-3000 CCRL rated engine to compare. I don't do LTC, so 3-minute games are the bottom limit for me, which should be fast enough, and then run a gauntlet of ... 200+ games each.

I kind of missed what your suggested course of action was for Leela Zero. I mean concrete and precise, not some waving of the hands with how things 'need to be changed'.

[David Xu]

The Lella Chess project is trying to "replicate" the Alpha Zero project.
The Alpha Zero project is based on very different concepts from classical AB engines and they did not use AB engines to train the networks.
Remember the A0 team isn't stupid nor ignorant, they are pretty smart.
The A0 team proved (at least to me) that their methods really works, you just need to implement them very well.
The LC0 project is also proving these methods works. The main problem of LC0 is in my opinion the hardware. The Google team has access to fabulous hardware while the LC0 team don't, so they will have to wait quite a bit longer to train the networks. Unfortunately also during game play it is very limited in nodes per second, maybe in the future someone will aqquire some of those TPUS and test LC0 on that hardware. While you can run a classical AB engine on your PC and get very powerfull play from it you won't be able to get high performance from LC0 on a "normal" PC, at least on mine with a Radeon 4600 HD and a core i7 4 core Nehalem I won't get anywhere with LC0.

The problem is the following. AG0 like AG is based on MCTS which was the only successful concept in Go. So there was nothing new (at least in AG case) except monstrous hardware. With AG0 they introduced reinforcement learning. This is also fine, because their game model is Go where we have known for quite a while (well before Google got its hands on it) that MCTS+NN approach works. Additionally to reinforcement learning, in AG0 they unified policy and value network in one as another novelty. And we can say this has been reasonably demonstrated that it works for Go.
Chess is a different game altogether and just blindly assuming the concepts from Go would work equally well without proof is plain wrong.
Some take that lousy preprint as a proof, some are intimidated by Google and would take whatever they say at face value, fine, but I disagree with that.
We don't know if MCTS (or UCT in what ever version) would work for chess (there are strong indications it doesn't). We don't know if combining policy and value network in one NN would work for chess (so far judging by atrocious LC0 tactical performance it doesn't).
Yes LC0 is based on L0 simply because A0 is based on AG0. And what we know so far (beside awful tactical performance and very strong positional performance of LC0):
1) Training is saturated (it is a current state no matter how much some ppl bury their head in send). Whether it is because of bug in training code, small network size, bad training process, or simply too shallow training games due to the lack of monstrous hardware that Google had at their disposal is to be seen.
2) Despite having relatively large net (half of the A0) performance based on extrapolation to equal hardware as A0 (4TPUs) is 300-400Elo below A0. And we know that increasing net size brings initially very little if anything because of much slower evaluation. Doubling net size usually slows down nps for factor 2.5-3. When going from 6x64 to 10x128 Lc0 gained a lot. When going from 10x128 to 15x192 much, much less, and I'm talking here about testing with fixed number of playouts.
3) Scaling performance is not nearly as good as some expected it would be (ofc I would say) and is totally different to what has been presented in that lousy preprint. It is 50-60 Elo per doubling of playouts.

Nothing more than the usual special pleading from Milos. "Oh, it works for Go, but my pet board game must be different in a special way that makes all the techniques that worked for Go invalid!"

There is absolutely zero evidence to suggest that chess differs from Go in a domain-relevant way, and a general-purpose algorithm ought to be able to learn both equally well. It would be a surprise indeed to learn that a neural network is somehow less capable of developing chess understanding than Go understanding, and all the arguments to that effect consist mostly of vague handwaving. Come back when you have actual data.

[Daniel Shawul]

Nothing more than the usual special pleading from Milos. "Oh, it works for Go, but my pet board game must be different in a special way that makes all the techniques that worked for Go invalid!"

There is absolutely zero evidence to suggest that chess differs from Go in a domain-relevant way, and a general-purpose algorithm ought to be able to learn both equally well.

This demonstrates you have 0 understanding of the problem. Before A0, MCTS was never remotely as successful on chess (and for that matter in many other games) as it was on Go. There are _tons_ of papers on the problems of MCTS and chess that you must be leaving on mars to not stumble upon one.

It would be a surprise indeed to learn that a neural network is somehow less capable of developing chess understanding than Go understanding, and all the arguments to that effect consist mostly of vague handwaving. Come back when you have actual data.

Actually Milos summarized the issue very well, but you, on the other hand, need to know the basics with MCTS, and NN+tactics before opening your mouth.

[jp]

Before A0, MCTS was never remotely as successful on chess (and for that matter in many other games) as it was on Go. There are _tons_ of papers on the problems of MCTS and chess that you must be leaving on mars to not stumble upon one.

Could you give one or two of the papers that you regard as the greatest cause for pessimism?

[Daniel Shawul]

Before A0, MCTS was never remotely as successful on chess (and for that matter in many other games) as it was on Go. There are _tons_ of papers on the problems of MCTS and chess that you must be leaving on mars to not stumble upon one.

Could you give one or two of the papers that you regard as the greatest cause for pessimism?

Google is your friend.

For such loud mouth "defenders of A0", you lack basic understanding of why people are questioning their results.

[jp]

Google is your friend.

For such loud mouth "defenders of A0", you lack basic understanding of why people are questioning their results.

Daniel Shawul, you replied to my neutral post.
The person who you think is a "loud mouth defender" is someone else, David Xu.
We are different people with different opinions.

[Milos]

Google is your friend.

For such loud mouth "defenders of A0", you lack basic understanding of why people are questioning their results.

Daniel Shawul, you replied to my neutral post.
The person who you think is a "loud mouth defender" is someone else, David Xu.
We are different people with different opinions.

Daniel probably thought you are just being provocative since topic is really well explored (therefore even less understanding for David Xu and his ignorance) and it is trivial to find references using Google, but I will anyway help you.
Look at MCTS and UCT, in both under section References.
More specifically you can pay attention to the relatively recent paper that Daniel already discussed on this forum.