LC0 on the WAC test suite, bad results

[jkiliani]

I am dumbfounded by people who think its tactics could be improved by training. Tactics is something that is dynamic and should be evaluated precisely; not something suitable for a universal approximator like a NN. If you train on WAC it is not going to help you anywhere else. My guess is its tactics is gonna suck forever unless elements of alpha-beta are introduced such as, using minmax backups, alpha-beta rollouts etc..

About the 3x3 filters, why does LCzero have code for inference anyway ? My first take with NN chess engine would probably use the c++ tensorflow backend for inference, which would use cuDNN as its backend which probably has better optimized algorithms than hand-written ones. Is this done for to avoid dependency on tensorflow or am I missing something ?

Daniel

We'll see about the tactics once the network size is much larger than now, your guess is as good as mine. About the inference code, LCZero is using the same OpenCL code as Leela Zero here, mainly for the reason that Nvidia has very restrictive licensing on CuDNN. Ask Gian-Carlo about his opinion on that sometime...

[Milos]

mainly for the reason that Nvidia has very restrictive licensing on CuDNN. Ask Gian-Carlo about his opinion on that sometime...

I think you heavily misunderstood something.
There is no other restrictiveness on CuDNN except portability.
It is used at University, in open source projects, commercially (in my company we use it without problem to compile code that goes into products without any additional license agreement with NVIDIA), you name it.
OpenCL is convenient because you call the same library no matter what is your target platform and that it enables heterogeneity. And some ppl (very stupid) religious attitudes that everything they use have to be open source.
However, 3x3 handwritten convolutional filters are 2x slower even with best possibly tuned handwritten OpenCL implementation which can be clearly seen from that tutorial that Gian-Carlo posted.
Similar thing happens if you use OpenBlas with CPUs that have AVX512. You get more than factor of 2 slower performance compared to IntelMKL.

[Robert Pope]

mainly for the reason that Nvidia has very restrictive licensing on CuDNN. Ask Gian-Carlo about his opinion on that sometime...

I think you heavily misunderstood something.
There is no other restrictiveness on CuDNN except portability.
It is used at University, in open source projects, commercially (in my company we use it without problem to compile code that goes into products without any additional license agreement with NVIDIA), you name it.
OpenCL is convenient because you call the same library no matter what is your target platform and that it enables heterogeneity. And some ppl (very stupid) religious attitudes that everything they use have to be open source.
However, 3x3 handwritten convolutional filters are 2x slower even with best possibly tuned handwritten OpenCL implementation which can be clearly seen from that tutorial that Gian-Carlo posted.
Similar thing happens if you use OpenBlas with CPUs that have AVX512. You get more than factor of 2 slower performance compared to IntelMKL.

Can you point me to the tutorial you mention here?

[Dann Corbit]

At some point in the future, it could probably be tried to train Leela (or a fork of her) on a mix of self-play games and tactical suites like this one. There's a chance that this would make the net at least more familiar with these types of positions, which is what's required to solve such puzzles. May take quite a bit of trial and error though to do that without weakening the general gameplay of the neural net, and it will definitely require a very LARGE database of tactical puzzles to train on.

I am dumbfounded by people who think its tactics could be improved by training. Tactics is something that is dynamic and should be evaluated precisely; not something suitable for a universal approximator like a NN. If you train on WAC it is not going to help you anywhere else. My guess is its tactics is gonna suck forever unless elements of alpha-beta are introduced such as, using minmax backups, alpha-beta rollouts etc..

About the 3x3 filters, why does LCzero have code for inference anyway ? My first take with NN chess engine would probably use the c++ tensorflow backend for inference, which would use cuDNN as its backend which probably has better optimized algorithms than hand-written ones. Is this done for to avoid dependency on tensorflow or am I missing something ?

Daniel

If neural nets work like human neural nets, then I would expect that tactics can be learned also.

A human can recognize many tactics by pattern recognition.
Simple example. KR vs K, instantly recognized as a win and the algorithm for winning is also imprinted in the brain to play without even thinking.

Another simple example: unstoppable passed pawn. Once you see that you have one, you know you have an enormous club on your shoulder without even having to play it out. So a computer can recognize things like this also.

Surely there are tactics that only become apparent after alpha-beta without extra pruning. But I guess that these are very rare.

When we play chess we recognize configurations that have tactical outcome.

Now, I do not know that the neural nets we make operate like human neural nets. But if they do, they can learn tactics just like we do.

[Daniel Shawul]

At some point in the future, it could probably be tried to train Leela (or a fork of her) on a mix of self-play games and tactical suites like this one. There's a chance that this would make the net at least more familiar with these types of positions, which is what's required to solve such puzzles. May take quite a bit of trial and error though to do that without weakening the general gameplay of the neural net, and it will definitely require a very LARGE database of tactical puzzles to train on.

I am dumbfounded by people who think its tactics could be improved by training. Tactics is something that is dynamic and should be evaluated precisely; not something suitable for a universal approximator like a NN. If you train on WAC it is not going to help you anywhere else. My guess is its tactics is gonna suck forever unless elements of alpha-beta are introduced such as, using minmax backups, alpha-beta rollouts etc..

About the 3x3 filters, why does LCzero have code for inference anyway ? My first take with NN chess engine would probably use the c++ tensorflow backend for inference, which would use cuDNN as its backend which probably has better optimized algorithms than hand-written ones. Is this done for to avoid dependency on tensorflow or am I missing something ?

Daniel

If neural nets work like human neural nets, then I would expect that tactics can be learned also.

A human can recognize many tactics by pattern recognition.
Simple example. KR vs K, instantly recognized as a win and the algorithm for winning is also imprinted in the brain to play without even thinking.

Another simple example: unstoppable passed pawn. Once you see that you have one, you know you have an enormous club on your shoulder without even having to play it out. So a computer can recognize things like this also.

Surely there are tactics that only become apparent after alpha-beta without extra pruning. But I guess that these are very rare.

When we play chess we recognize configurations that have tactical outcome.

Now, I do not know that the neural nets we make operate like human neural nets. But if they do, they can learn tactics just like we do.

But a human brain can also do a lookahead search, though a very selective one at that. The combined MCTS+NN model is the one comparable to how the human brain thinks in chess. Taking one quick look at a position you can not tell right away of the tactical outcome (not positional). I am pretty sure the NN alone can't do it, so the debate was if the policy NN can guide the MCTS search precisely to solve tactics. The problem with this idea is that whether the policy NN is good or bad, it is a conundrum, you are going to suffer from tactics anyway. If it is good, it won't find tactical traps (bad looking moves turning out to be good on further exhaustive search), and if it is bad, well, it is bad.

The alphabeta+eval combination is a typical "computer" model that will exhaustively look at all the possibilities and will not miss tactics in its horizon. All the standard chess engines are designed not to miss tactics and sniff out tactical mistakes of the opposition. I have no explanation for AlphaZero's dominance given this weakness of MCTS, and that stockfish is a tactical monster. Maybe the 180x hardware advantage has something to do with it.

With its current approach, lczero tromps oppostions on positonal understanding but suffers from tactics.

Daniel

[Dann Corbit]

{snip}
With its current approach, lczero tromps oppostions on positonal understanding but suffers from tactics.

Daniel

Interesting.

Remember, way back in the olden days, when we tuned Scorpio for tactics using parabolic fits? I wonder if a SF engine, tuned with that technique, would have extraordinary success against not only LCZero but also AlphaZero.

Of course, the positional play suffers, but it would be a tactical trouble maker, for sure.

[Milos]

mainly for the reason that Nvidia has very restrictive licensing on CuDNN. Ask Gian-Carlo about his opinion on that sometime...

I think you heavily misunderstood something.
There is no other restrictiveness on CuDNN except portability.
It is used at University, in open source projects, commercially (in my company we use it without problem to compile code that goes into products without any additional license agreement with NVIDIA), you name it.
OpenCL is convenient because you call the same library no matter what is your target platform and that it enables heterogeneity. And some ppl (very stupid) religious attitudes that everything they use have to be open source.
However, 3x3 handwritten convolutional filters are 2x slower even with best possibly tuned handwritten OpenCL implementation which can be clearly seen from that tutorial that Gian-Carlo posted.
Similar thing happens if you use OpenBlas with CPUs that have AVX512. You get more than factor of 2 slower performance compared to IntelMKL.

Can you point me to the tutorial you mention here?

https://cnugteren.github.io/tutorial/pages/page1.html

[mirek]

If neural nets work like human neural nets, then I would expect that tactics can be learned also.

A human can recognize many tactics by pattern recognition.
Simple example. KR vs K, instantly recognized as a win and the algorithm for winning is also imprinted in the brain to play without even thinking.

Another simple example: unstoppable passed pawn. Once you see that you have one, you know you have an enormous club on your shoulder without even having to play it out. So a computer can recognize things like this also.

Surely there are tactics that only become apparent after alpha-beta without extra pruning. But I guess that these are very rare.

When we play chess we recognize configurations that have tactical outcome.

Now, I do not know that the neural nets we make operate like human neural nets. But if they do, they can learn tactics just like we do.

But a human brain can also do a lookahead search, though a very selective one at that. The combined MCTS+NN model is the one comparable to how the human brain thinks in chess. Taking one quick look at a position you can not tell right away of the tactical outcome (not positional). I am pretty sure the NN alone can't do it, so the debate was if the policy NN can guide the MCTS search precisely to solve tactics. The problem with this idea is that whether the policy NN is good or bad, it is a conundrum, you are going to suffer from tactics anyway. If it is good, it won't find tactical traps (bad looking moves turning out to be good on further exhaustive search), and if it is bad, well, it is bad.

Once again: how do human players find candidate moves? I think it should be obvious that it's based on pattern recognition and it's very similar to policy NN of LC0 / A0. Humans definitely do not play chess by exhausting all variations within given depth.

So if I would rephrase your sentence with biological NN in mind:

The problem with this idea is that whether human's brain policy network is good or bad, it is a conundrum, you are going to suffer from tactics anyway. If it is good, it won't find tactical traps (bad looking moves turning out to be good on further exhaustive search), and if it is bad, well, it is bad.

In other words: human players are incapable of finding / setting or recognizing traps? Is this what you are saying?

I doubt that - so perhaps you are saying it won't find ALL the traps that can be in the position ALL the time? But that is the same as with all other tactics. Even well trained policy NN driven search may overlook some tactics which would be found by A/B exhaustive search. And despite all that the NN can actually play stronger chess. But we have already discussed effect of lowering branching factor and why it is much more useful than always finding all tactical traps. But if it wasn't enough here is still another counterexample

Just consider the limiting case of "pure tactical engine" which basically means it's capable of finding all forced checkmates within the given depth while evaluating all other moves as equal. Now lets' say that it would be capable to see all forced checkmates until move 40 - so much deeper than any "standard" engine is capable of seeing in blitz.
Now let this perfect 40 moves ahead "tactical monster" play blitz against any half-decent normal engine and I am pretty sure that the "tactical monster" would lose pretty badly. Exhaustive "perfect tactics" 40 moves ahead is useless. In reality good balance of tactics + strategy is what gives elo. Case closed?

[Uri Blass]

If neural nets work like human neural nets, then I would expect that tactics can be learned also.

A human can recognize many tactics by pattern recognition.
Simple example. KR vs K, instantly recognized as a win and the algorithm for winning is also imprinted in the brain to play without even thinking.

Another simple example: unstoppable passed pawn. Once you see that you have one, you know you have an enormous club on your shoulder without even having to play it out. So a computer can recognize things like this also.

Surely there are tactics that only become apparent after alpha-beta without extra pruning. But I guess that these are very rare.

When we play chess we recognize configurations that have tactical outcome.

Now, I do not know that the neural nets we make operate like human neural nets. But if they do, they can learn tactics just like we do.

But a human brain can also do a lookahead search, though a very selective one at that. The combined MCTS+NN model is the one comparable to how the human brain thinks in chess. Taking one quick look at a position you can not tell right away of the tactical outcome (not positional). I am pretty sure the NN alone can't do it, so the debate was if the policy NN can guide the MCTS search precisely to solve tactics. The problem with this idea is that whether the policy NN is good or bad, it is a conundrum, you are going to suffer from tactics anyway. If it is good, it won't find tactical traps (bad looking moves turning out to be good on further exhaustive search), and if it is bad, well, it is bad.

Once again: how do human players find candidate moves? I think it should be obvious that it's based on pattern recognition and it's very similar to policy NN of LC0 / A0. Humans definitely do not play chess by exhausting all variations within given depth.

I disagree.
Human players can find candidate moves also based on searching all moves one ply forward.

I decided in the past(in OTB long time control games) to try not to play a move before I looked one ply forward at other options I could play or at the possible moves of the opponent against the move I planned.
I know that I got a significant improvement in chess in the past exactly from this decision that helped to improve my thinking process.

This decision was because I noticed that significant part of my mistakes were because of not considering the next move of myself or the next move of my opponent even for a minimal time of 1 second.

It was a significant part in my improvement from a rating of 1600 at tournament time control to higher rating.

An engine that does not try to do brute force search even to small depth simply does not play chess in the same way that smart humans play.

Note that I am not saying that brute force to small depth should be all the search but it should be at least part of the search.

[Evert]

Once again: how do human players find candidate moves? I think it should be obvious that it's based on pattern recognition and it's very similar to policy NN of LC0 / A0. Humans definitely do not play chess by exhausting all variations within given depth.

So if I would rephrase your sentence with biological NN in mind:

The problem with this idea is that whether human's brain policy network is good or bad, it is a conundrum, you are going to suffer from tactics anyway. If it is good, it won't find tactical traps (bad looking moves turning out to be good on further exhaustive search), and if it is bad, well, it is bad.

In other words: human players are incapable of finding / setting or recognizing traps? Is this what you are saying?

It should be remembered of course that alpha-beta searchers perform better at chess than humans do. So it's not so obvious that modelling the way humans think would produce the strongest chess program.
But it seems to me that many people have incorrect expectations or questions about the Leela project.
So here's my take on a few of the discussion points I see coming up repeatedly.
Can the neural net learn to recognise tactics? Probably; it could learn the equivalent of some elaborate SEE function. Will that catch everything all the time? Probably not.
Will a neural net learn to play optimal chess (for some definition of optimal)? Probably not, and certainly not if it is rewarded for winning rather than winning quickly.
Is a minimax backup a better idea than an average backup? Almost certainly. It's also what humans do, but of course I just said copying what humans do might not be a good idea.
Is a general setup that has no input neurons associated with features that are specifically relevant for chess optimal? Almost certainly not.
Is starting with something completely general and then testing whether you can get strong play and certain patterns from emergent knowledge an interesting idea? Certainly. However, it makes sense to me to start with something more specific (which is what Google did with AlphaGo). I have to say though, I love emergent phenomena.

So I suspect that if your goal is to build the strongest neural network chess engine, then Leela is not the optimal way to do it. That doesn't mean it's not an interesting or worthwhile approach.