LC0 on the WAC test suite, bad results

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
jkiliani
Posts: 143
Joined: Wed Jan 17, 2018 12:26 pm

Re: LC0 on the WAC test suite, bad results

Post by jkiliani » Thu Apr 26, 2018 8:04 pm

Daniel Shawul wrote:I am dumbfounded by people who think its tactics could be improved by training. Tactics is something that is dynamic and should be evaluated precisely; not something suitable for a universal approximator like a NN. If you train on WAC it is not going to help you anywhere else. My guess is its tactics is gonna suck forever unless elements of alpha-beta are introduced such as, using minmax backups, alpha-beta rollouts etc..

About the 3x3 filters, why does LCzero have code for inference anyway ? My first take with NN chess engine would probably use the c++ tensorflow backend for inference, which would use cuDNN as its backend which probably has better optimized algorithms than hand-written ones. Is this done for to avoid dependency on tensorflow or am I missing something ?

Daniel
We'll see about the tactics once the network size is much larger than now, your guess is as good as mine. About the inference code, LCZero is using the same OpenCL code as Leela Zero here, mainly for the reason that Nvidia has very restrictive licensing on CuDNN. Ask Gian-Carlo about his opinion on that sometime...

Milos
Posts: 3387
Joined: Wed Nov 25, 2009 12:47 am

Re: LC0 on the WAC test suite, bad results

Post by Milos » Thu Apr 26, 2018 8:46 pm

jkiliani wrote:mainly for the reason that Nvidia has very restrictive licensing on CuDNN. Ask Gian-Carlo about his opinion on that sometime...
I think you heavily misunderstood something.
There is no other restrictiveness on CuDNN except portability.
It is used at University, in open source projects, commercially (in my company we use it without problem to compile code that goes into products without any additional license agreement with NVIDIA), you name it.
OpenCL is convenient because you call the same library no matter what is your target platform and that it enables heterogeneity. And some ppl (very stupid) religious attitudes that everything they use have to be open source.
However, 3x3 handwritten convolutional filters are 2x slower even with best possibly tuned handwritten OpenCL implementation which can be clearly seen from that tutorial that Gian-Carlo posted.
Similar thing happens if you use OpenBlas with CPUs that have AVX512. You get more than factor of 2 slower performance compared to IntelMKL.

Robert Pope
Posts: 510
Joined: Sat Mar 25, 2006 7:27 pm

Re: LC0 on the WAC test suite, bad results

Post by Robert Pope » Thu Apr 26, 2018 8:56 pm

Milos wrote:
jkiliani wrote:mainly for the reason that Nvidia has very restrictive licensing on CuDNN. Ask Gian-Carlo about his opinion on that sometime...
I think you heavily misunderstood something.
There is no other restrictiveness on CuDNN except portability.
It is used at University, in open source projects, commercially (in my company we use it without problem to compile code that goes into products without any additional license agreement with NVIDIA), you name it.
OpenCL is convenient because you call the same library no matter what is your target platform and that it enables heterogeneity. And some ppl (very stupid) religious attitudes that everything they use have to be open source.
However, 3x3 handwritten convolutional filters are 2x slower even with best possibly tuned handwritten OpenCL implementation which can be clearly seen from that tutorial that Gian-Carlo posted.
Similar thing happens if you use OpenBlas with CPUs that have AVX512. You get more than factor of 2 slower performance compared to IntelMKL.
Can you point me to the tutorial you mention here?

Dann Corbit
Posts: 10192
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: LC0 on the WAC test suite, bad results

Post by Dann Corbit » Thu Apr 26, 2018 9:29 pm

Daniel Shawul wrote:
jkiliani wrote:At some point in the future, it could probably be tried to train Leela (or a fork of her) on a mix of self-play games and tactical suites like this one. There's a chance that this would make the net at least more familiar with these types of positions, which is what's required to solve such puzzles. May take quite a bit of trial and error though to do that without weakening the general gameplay of the neural net, and it will definitely require a very LARGE database of tactical puzzles to train on.
I am dumbfounded by people who think its tactics could be improved by training. Tactics is something that is dynamic and should be evaluated precisely; not something suitable for a universal approximator like a NN. If you train on WAC it is not going to help you anywhere else. My guess is its tactics is gonna suck forever unless elements of alpha-beta are introduced such as, using minmax backups, alpha-beta rollouts etc..

About the 3x3 filters, why does LCzero have code for inference anyway ? My first take with NN chess engine would probably use the c++ tensorflow backend for inference, which would use cuDNN as its backend which probably has better optimized algorithms than hand-written ones. Is this done for to avoid dependency on tensorflow or am I missing something ?

Daniel
If neural nets work like human neural nets, then I would expect that tactics can be learned also.

A human can recognize many tactics by pattern recognition.
Simple example. KR vs K, instantly recognized as a win and the algorithm for winning is also imprinted in the brain to play without even thinking.

Another simple example: unstoppable passed pawn. Once you see that you have one, you know you have an enormous club on your shoulder without even having to play it out. So a computer can recognize things like this also.

Surely there are tactics that only become apparent after alpha-beta without extra pruning. But I guess that these are very rare.

When we play chess we recognize configurations that have tactical outcome.

Now, I do not know that the neural nets we make operate like human neural nets. But if they do, they can learn tactics just like we do.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

Daniel Shawul
Posts: 3762
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: LC0 on the WAC test suite, bad results

Post by Daniel Shawul » Thu Apr 26, 2018 9:47 pm

Dann Corbit wrote:
Daniel Shawul wrote:
jkiliani wrote:At some point in the future, it could probably be tried to train Leela (or a fork of her) on a mix of self-play games and tactical suites like this one. There's a chance that this would make the net at least more familiar with these types of positions, which is what's required to solve such puzzles. May take quite a bit of trial and error though to do that without weakening the general gameplay of the neural net, and it will definitely require a very LARGE database of tactical puzzles to train on.
I am dumbfounded by people who think its tactics could be improved by training. Tactics is something that is dynamic and should be evaluated precisely; not something suitable for a universal approximator like a NN. If you train on WAC it is not going to help you anywhere else. My guess is its tactics is gonna suck forever unless elements of alpha-beta are introduced such as, using minmax backups, alpha-beta rollouts etc..

About the 3x3 filters, why does LCzero have code for inference anyway ? My first take with NN chess engine would probably use the c++ tensorflow backend for inference, which would use cuDNN as its backend which probably has better optimized algorithms than hand-written ones. Is this done for to avoid dependency on tensorflow or am I missing something ?

Daniel
If neural nets work like human neural nets, then I would expect that tactics can be learned also.

A human can recognize many tactics by pattern recognition.
Simple example. KR vs K, instantly recognized as a win and the algorithm for winning is also imprinted in the brain to play without even thinking.

Another simple example: unstoppable passed pawn. Once you see that you have one, you know you have an enormous club on your shoulder without even having to play it out. So a computer can recognize things like this also.

Surely there are tactics that only become apparent after alpha-beta without extra pruning. But I guess that these are very rare.

When we play chess we recognize configurations that have tactical outcome.

Now, I do not know that the neural nets we make operate like human neural nets. But if they do, they can learn tactics just like we do.
But a human brain can also do a lookahead search, though a very selective one at that. The combined MCTS+NN model is the one comparable to how the human brain thinks in chess. Taking one quick look at a position you can not tell right away of the tactical outcome (not positional). I am pretty sure the NN alone can't do it, so the debate was if the policy NN can guide the MCTS search precisely to solve tactics. The problem with this idea is that whether the policy NN is good or bad, it is a conundrum, you are going to suffer from tactics anyway. If it is good, it won't find tactical traps (bad looking moves turning out to be good on further exhaustive search), and if it is bad, well, it is bad.

The alphabeta+eval combination is a typical "computer" model that will exhaustively look at all the possibilities and will not miss tactics in its horizon. All the standard chess engines are designed not to miss tactics and sniff out tactical mistakes of the opposition. I have no explanation for AlphaZero's dominance given this weakness of MCTS, and that stockfish is a tactical monster. Maybe the 180x hardware advantage has something to do with it.

With its current approach, lczero tromps oppostions on positonal understanding but suffers from tactics.

Daniel

Dann Corbit
Posts: 10192
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: LC0 on the WAC test suite, bad results

Post by Dann Corbit » Thu Apr 26, 2018 9:50 pm

Daniel Shawul wrote: {snip}
With its current approach, lczero tromps oppostions on positonal understanding but suffers from tactics.

Daniel
Interesting.

Remember, way back in the olden days, when we tuned Scorpio for tactics using parabolic fits? I wonder if a SF engine, tuned with that technique, would have extraordinary success against not only LCZero but also AlphaZero.

Of course, the positional play suffers, but it would be a tactical trouble maker, for sure.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

Milos
Posts: 3387
Joined: Wed Nov 25, 2009 12:47 am

Re: LC0 on the WAC test suite, bad results

Post by Milos » Thu Apr 26, 2018 10:24 pm

Robert Pope wrote:
Milos wrote:
jkiliani wrote:mainly for the reason that Nvidia has very restrictive licensing on CuDNN. Ask Gian-Carlo about his opinion on that sometime...
I think you heavily misunderstood something.
There is no other restrictiveness on CuDNN except portability.
It is used at University, in open source projects, commercially (in my company we use it without problem to compile code that goes into products without any additional license agreement with NVIDIA), you name it.
OpenCL is convenient because you call the same library no matter what is your target platform and that it enables heterogeneity. And some ppl (very stupid) religious attitudes that everything they use have to be open source.
However, 3x3 handwritten convolutional filters are 2x slower even with best possibly tuned handwritten OpenCL implementation which can be clearly seen from that tutorial that Gian-Carlo posted.
Similar thing happens if you use OpenBlas with CPUs that have AVX512. You get more than factor of 2 slower performance compared to IntelMKL.
Can you point me to the tutorial you mention here?
https://cnugteren.github.io/tutorial/pages/page1.html

mirek
Posts: 42
Joined: Sat Mar 24, 2018 3:18 pm

Re: LC0 on the WAC test suite, bad results

Post by mirek » Thu Apr 26, 2018 11:23 pm

Daniel Shawul wrote:
Dann Corbit wrote: If neural nets work like human neural nets, then I would expect that tactics can be learned also.

A human can recognize many tactics by pattern recognition.
Simple example. KR vs K, instantly recognized as a win and the algorithm for winning is also imprinted in the brain to play without even thinking.

Another simple example: unstoppable passed pawn. Once you see that you have one, you know you have an enormous club on your shoulder without even having to play it out. So a computer can recognize things like this also.

Surely there are tactics that only become apparent after alpha-beta without extra pruning. But I guess that these are very rare.

When we play chess we recognize configurations that have tactical outcome.

Now, I do not know that the neural nets we make operate like human neural nets. But if they do, they can learn tactics just like we do.
But a human brain can also do a lookahead search, though a very selective one at that. The combined MCTS+NN model is the one comparable to how the human brain thinks in chess. Taking one quick look at a position you can not tell right away of the tactical outcome (not positional). I am pretty sure the NN alone can't do it, so the debate was if the policy NN can guide the MCTS search precisely to solve tactics. The problem with this idea is that whether the policy NN is good or bad, it is a conundrum, you are going to suffer from tactics anyway. If it is good, it won't find tactical traps (bad looking moves turning out to be good on further exhaustive search), and if it is bad, well, it is bad.
Once again: how do human players find candidate moves? I think it should be obvious that it's based on pattern recognition and it's very similar to policy NN of LC0 / A0. Humans definitely do not play chess by exhausting all variations within given depth.

So if I would rephrase your sentence with biological NN in mind:

The problem with this idea is that whether human's brain policy network is good or bad, it is a conundrum, you are going to suffer from tactics anyway. If it is good, it won't find tactical traps (bad looking moves turning out to be good on further exhaustive search), and if it is bad, well, it is bad.

In other words: human players are incapable of finding / setting or recognizing traps? Is this what you are saying?

I doubt that - so perhaps you are saying it won't find ALL the traps that can be in the position ALL the time? But that is the same as with all other tactics. Even well trained policy NN driven search may overlook some tactics which would be found by A/B exhaustive search. And despite all that the NN can actually play stronger chess. But we have already discussed effect of lowering branching factor and why it is much more useful than always finding all tactical traps. But if it wasn't enough here is still another counterexample :lol:

Just consider the limiting case of "pure tactical engine" which basically means it's capable of finding all forced checkmates within the given depth while evaluating all other moves as equal. Now lets' say that it would be capable to see all forced checkmates until move 40 - so much deeper than any "standard" engine is capable of seeing in blitz.
Now let this perfect 40 moves ahead "tactical monster" play blitz against any half-decent normal engine and I am pretty sure that the "tactical monster" would lose pretty badly. Exhaustive "perfect tactics" 40 moves ahead is useless. In reality good balance of tactics + strategy is what gives elo. Case closed?

Uri Blass
Posts: 8605
Joined: Wed Mar 08, 2006 11:37 pm
Location: Tel-Aviv Israel

Re: LC0 on the WAC test suite, bad results

Post by Uri Blass » Fri Apr 27, 2018 4:34 am

mirek wrote:
Daniel Shawul wrote:
Dann Corbit wrote: If neural nets work like human neural nets, then I would expect that tactics can be learned also.

A human can recognize many tactics by pattern recognition.
Simple example. KR vs K, instantly recognized as a win and the algorithm for winning is also imprinted in the brain to play without even thinking.

Another simple example: unstoppable passed pawn. Once you see that you have one, you know you have an enormous club on your shoulder without even having to play it out. So a computer can recognize things like this also.

Surely there are tactics that only become apparent after alpha-beta without extra pruning. But I guess that these are very rare.

When we play chess we recognize configurations that have tactical outcome.

Now, I do not know that the neural nets we make operate like human neural nets. But if they do, they can learn tactics just like we do.
But a human brain can also do a lookahead search, though a very selective one at that. The combined MCTS+NN model is the one comparable to how the human brain thinks in chess. Taking one quick look at a position you can not tell right away of the tactical outcome (not positional). I am pretty sure the NN alone can't do it, so the debate was if the policy NN can guide the MCTS search precisely to solve tactics. The problem with this idea is that whether the policy NN is good or bad, it is a conundrum, you are going to suffer from tactics anyway. If it is good, it won't find tactical traps (bad looking moves turning out to be good on further exhaustive search), and if it is bad, well, it is bad.
Once again: how do human players find candidate moves? I think it should be obvious that it's based on pattern recognition and it's very similar to policy NN of LC0 / A0. Humans definitely do not play chess by exhausting all variations within given depth.
I disagree.
Human players can find candidate moves also based on searching all moves one ply forward.

I decided in the past(in OTB long time control games) to try not to play a move before I looked one ply forward at other options I could play or at the possible moves of the opponent against the move I planned.
I know that I got a significant improvement in chess in the past exactly from this decision that helped to improve my thinking process.

This decision was because I noticed that significant part of my mistakes were because of not considering the next move of myself or the next move of my opponent even for a minimal time of 1 second.

It was a significant part in my improvement from a rating of 1600 at tournament time control to higher rating.

An engine that does not try to do brute force search even to small depth simply does not play chess in the same way that smart humans play.

Note that I am not saying that brute force to small depth should be all the search but it should be at least part of the search.

User avatar
Evert
Posts: 2923
Joined: Fri Jan 21, 2011 11:42 pm
Location: NL
Contact:

Re: LC0 on the WAC test suite, bad results

Post by Evert » Fri Apr 27, 2018 5:28 am

mirek wrote: Once again: how do human players find candidate moves? I think it should be obvious that it's based on pattern recognition and it's very similar to policy NN of LC0 / A0. Humans definitely do not play chess by exhausting all variations within given depth.

So if I would rephrase your sentence with biological NN in mind:

The problem with this idea is that whether human's brain policy network is good or bad, it is a conundrum, you are going to suffer from tactics anyway. If it is good, it won't find tactical traps (bad looking moves turning out to be good on further exhaustive search), and if it is bad, well, it is bad.

In other words: human players are incapable of finding / setting or recognizing traps? Is this what you are saying?
It should be remembered of course that alpha-beta searchers perform better at chess than humans do. So it's not so obvious that modelling the way humans think would produce the strongest chess program.
But it seems to me that many people have incorrect expectations or questions about the Leela project.
So here's my take on a few of the discussion points I see coming up repeatedly.
Can the neural net learn to recognise tactics? Probably; it could learn the equivalent of some elaborate SEE function. Will that catch everything all the time? Probably not.
Will a neural net learn to play optimal chess (for some definition of optimal)? Probably not, and certainly not if it is rewarded for winning rather than winning quickly.
Is a minimax backup a better idea than an average backup? Almost certainly. It's also what humans do, but of course I just said copying what humans do might not be a good idea.
Is a general setup that has no input neurons associated with features that are specifically relevant for chess optimal? Almost certainly not.
Is starting with something completely general and then testing whether you can get strong play and certain patterns from emergent knowledge an interesting idea? Certainly. However, it makes sense to me to start with something more specific (which is what Google did with AlphaGo). I have to say though, I love emergent phenomena.

So I suspect that if your goal is to build the strongest neural network chess engine, then Leela is not the optimal way to do it. That doesn't mean it's not an interesting or worthwhile approach.

Post Reply