LCZero Accomplishments and Goals Thus Far

[jp]

I think Stockfish was crippled because they deprived it from using an opening-book, let it run on a 32 core machine with 64 threads (hyper-threading), used a fixed time per move and deprived it from using an egtb.

An opening book is not Stockfish. This is like saying I am crippled when playing a Chess game because I am not allowed to have Kasparov make the first 25 moves for me. If that counts as 'crippled', then AlphaZero was just as much crippled.

Some of these pro-AZ claims are very strange. What "is Stockfish" & your Kasparov argument is not a sensible argument. Your opponent won't allow you to get Kasparov to make your opening moves. But he cannot stop you memorizing opening theory Kasparov came up with.
You mentioned Formula I in another comment. Not giving an opening book is like not allowing the Formula I car to fuel before a race and claiming the solar car won the race. "The fuel is not Ferrari." So? "My solar car did not use fuel." So?
What matters is: did the solar car achieve great speed and distance etc.?
If the solar car guys boast about leaving behind the Ferrari that cannot move from the starting grid they are being silly or worse.
If the solar car does achieve good speed and distance, just be clear and open about exactly what speed and distance. Don't pretend it beat Ferrari in a Formula I race.


I was hoping we could stick to talking about Leela, not AZ...

[hgm]

Well, what I remembered was "Unless otherwise specified, the training and search algorithm and parameters are identical to AlphaGo Zero (29)." at the bottom of page 14, but on re-reading it one could argue that "algorithm" doesn't pertain to hardware.

It should also be obvious from the description of the training games that gen1 TPUs are used for playing, and the gen2 TPUs only for adjusting the NN. (In AlphaZero; AlphaGo Zero used GPU's for that.)

[jp]

...
It should also be obvious from the description of the training games that gen1 TPUs are used for playing, and the gen2 TPUs only for adjusting the NN. (In AlphaZero; AlphaGo Zero used GPU's for that.)

Yes, I got the impression it was gen1 TPUs for playing...

[David Xu]

Forgive me if I'm unsympathetic to claims of the form "If Stockfish had been allowed to have some other entity play moves for it at certain points during the game, it would have done better." Such claims are irrelevant to the match DeepMind ran, which was intended to test how the engines perform on their own, with no outside assistance.

[hgm]

Your opponent won't allow you to get Kasparov to make your opening moves. But he cannot stop you memorizing opening theory Kasparov came up with.

But Stockfish doesn't memorize anything. It relies on the GUI to play from the book in its place.

You mentioned Formula I in another comment. Not giving an opening book is like not allowing the Formula I car to fuel before a race and claiming the solar car won the race. "The fuel is not Ferrari." So? "My solar car did not use fuel." So?

The GP example pertained to algorithmic complexity, and had absolutely nothing to do with opening books.

What matters is: did the solar car achieve great speed and distance etc.?
If the solar car guys boast about leaving behind the Ferrari that cannot move from the starting grid they are being silly or worse.

And that was exactly what the metaphor was supposed to point out. You have to define a sensible metric, or you can arrive at any conclusion you want. If the metric is fuel efficiency, the solar car obviously wins. If the metric is "speed at all cost" the Ferrari obviously wins. Both metrics are silly, as can be seen from the fact that almost no one does buy a Ferrari or a Solar Car, and we all drive around in our Toyota's and Volkswagens. Likewise, applying as measure for algorithm improvement how well it performs on an x64 CPU is silly or worse.

I was hoping we could stick to talking about Leela, not AZ...

As LC0 and A0 are similar, issues that pertain to one, often pertain identically to the other. It doesn't really matter whether you are talking about a TPU or a GPU when showing the claims that these would provide a 'hardware advantage' are false.

[jp]

Forgive me if I'm unsympathetic to claims of the form "If Stockfish had been allowed to have some other entity play moves for it at certain points during the game, it would have done better."

As I said, I think it's better to talk about Leela...

But even your "unsympathetic" view should just lead you to conclude "Stockfish sucks at openings." Doesn't say much about Stockfish middlegames. Doesn't say much about other "entities", which are supposed to be the point, not what Stockfish does or does not suck at.

And that's actually just one of MANY issues that must have been discussed to death in previous threads.

[jp]

Your opponent won't allow you to get Kasparov to make your opening moves. But he cannot stop you memorizing opening theory Kasparov came up with.

But Stockfish doesn't memorize anything. It relies on the GUI to play from the book in its place.

...

I was hoping we could stick to talking about Leela, not AZ...

As LC0 and A0 are similar, issues that pertain to one, often pertain identically to the other. It doesn't really matter whether you are talking about a TPU or a GPU when showing the claims that these would provide a 'hardware advantage' are false.

No computer or computer program does anything except compute. They don't think. They are not conscious. I think most people accept that the closest equivalent to human memorizing in a computer is having a book, i.e. storing prior information in memory.

I think the big point about LC0 is that we can test it properly in any way we want with any metric we want on any hardware we want. We can do all of those things, and we can do it all in the correct way, report it all openly and objectively, & everyone can then be happy. That is the huge plus of having LC0.

[Daniel Shawul]

I am only interested in algorithmic improvements and in that regard to be afforded the luxury of acceleration of a bulky eval is very unfair.

But 'improvement' is a meaningless concept if you have no measure of quality. What would be an improvement in one metric, might be total degradation in another metric.

You see no algorithmic improvement in AlphaZero or LCZero, because you are using a completely flawed metric. Namely how well it performs on a single-core x64 CPU.

But no one cares about that. If you would use a more realistic metric, the neural network is a vast algorithmic improvement.

Almost everybody, even one's who used NN evals like Giraffe, cared about performance on single core upto now. It is simply the most widely used computing hardware now.
Graphics cards are mostly integrated ones on which LCzero performs equally or worse than the CPU. You are asking for a hardware that is available to gamers or people who do number crunching.

A question: Do you see an algorithmic improvement in DeepBlue's acceleration of its eval ? If not, what makes it different with A0's ? That GPU cards are easily accessible is irrelevant from an algorithmic point of view.

If you want to compare MCTS vs alpha-beta, you use the same eval for them and see who fairs better in chess/Go. Similarly, to compare evals, you use the same search, and see how well the NN eval fairs against the hand-made one on the same hardware.

If there is no hardware equivalence in comparing the evals the comparison is meaningless ofcourse as one can just add more knowledge without worry. Infact, a NN is probably the most inefficient tool for that, though it is a more generic one.
It waists too many FLOPs doing unnecessary multiplications for a not-so-important feature that a handmade eval would probably ignore. No one doubts (even without using NN) that you can increase your elo to your satisfaction by adding more knowledge -- the question is if can you get a better quality eval with the same FLOPs ?

[noobpwnftw]

To avoid cheap shots, engines should not provide configurable parameters about threading and stuff, only runs on custom laboratory hardware, as long as providing brief specs, probing an online up-to-date book and have TB requirements fused into the code. Without them, just exit().

I'm impressed by how many different ways people can come up with crippling engines and justify them as proper test methodology.

[jp]

...
I'm impressed by how many different ways people can come up with crippling engines and justify them as proper test methodology.

Yes, and probably even if they do not realize that that's what they are doing, i.e. this can happen even when it is not intentional.