LCZero Accomplishments and Goals Thus Far

[Joost Buijs]

Every scientist knows..., knowing is something else than doing. I've been working on a university for 26 years and I saw on many occasions that results were polished, cherrypicked or bad results left out to make the final outcome look more favourable. Not to speak about articles with totally fabricated results which also happens if you read the news.

This is known as "academic fraud", and it's a good way to get yourself fired. If you think Google DeepMind resorted to academic fraud, well, have fun with your conspiracy theorizing.

Meanwhile, the real world continues to move forward.

I don't think anything like that, I'm only saying that things not always have to be as 'rosy' as they look.

The whole development with respect to NN and deep-learning is interesting, but to call it a new concept is nonsense, all these techniques are known for years, in the past it was just not feasible because there existed no suitable hardware to cope with these things in a reasonable amount of time.

[hgm]

From the AlphaZero article it is not clear which TPU they used to play the games.

I didn't read the AlphaGo Zero article, and I don't get why the hardware of AlphaGo Zero and AlphaZero should be the same.

The AlphaZero paper only describes the differences with AlphaGo Zero, (and explicitly states that), so for everything that is not described there you will have to refer to the AlphaGo Zero paper.

In addition, it makes sense: the advantage of gen2 TPUs is that they can do floating point arithmetic, while the gen1 TPUs can only do 8 or (at half the speed) 16-bit multiplication. Floating-point arithmetic is only needed for the back-propagation during training of the network. To run the network on a position ('inference') the 8-bit precision is enough.

I think Stockfish was crippled because they deprived it from using an opening-book, let it run on a 32 core machine with 64 threads (hyper-threading), used a fixed time per move and deprived it from using an egtb.

An opening book is not Stockfish. This is like saying I am crippled when playing a Chess game because I am not allowed to have Kasparov make the first 25 moves for me. If that counts as 'crippled', then AlphaZero was just as much crippled. If your claim is that it is not proven that AlphaZero plays better opening lines that those known from centuries of collective effort of the entire human race, you might have a point. Not related to Stockfish, though. And no one was actually claiming that.

AlphaZero was not using EGTB (which do not provide measurable strength in a 100-game match anyway...). AlphaZero was using fixed time per move. That there are conditions under which Stockfish might perform better doesn't mean it is 'crippled'. If I play a 5-min blitz game, and lose, how do you think people would react in my Chess club if I kept complaining that I only lost because I was 'crippled' by having to complete the entire game in 5 min, while I play much stronger if I can think 5 min per move? I would be laughed out of the building...

As to the HT issue: it seems that Stockfish had a massive hardware advantage with 32 cores, while AlphaZero was only using 8 cores + 4 TPUs. So even if the hyer-threading was actually detrimental, how many threads non-HT cores would 64 HT be equivalent to? 24 cores? 16 cores? Still more than AlphaZero was using...

Every scientist knows..., knowing is something else than doing.

You are accusing them of plain, premeditated fraud. For virtually no gain, jeopardizing the great fame they achieved through their Go results (which cannot possibly have been forged). Does that sound likely to you? And yes, it does happen that millionaire movie stars are caught shop-lifting at the super-market. But so rarely that it makes international news, and they are considered mentally ill. How likely is it that a large team of people would want to ruin their future that way? I never heard of a gang of millionaires going around at night to burgle houses, which would roughly be the equivalent. Suspecting this sounds like complete paranoia.

[Daniel Shawul]

Every scientist knows..., knowing is something else than doing. I've been working on a university for 26 years and I saw on many occasions that results were polished, cherrypicked or bad results left out to make the final outcome look more favourable. Not to speak about articles with totally fabricated results which also happens if you read the news.

This is known as "academic fraud", and it's a good way to get yourself fired. If you think Google DeepMind resorted to academic fraud, well, have fun with your conspiracy theorizing.

Meanwhile, the real world continues to move forward.

I don't think anything like that, I'm only saying that things not always have to be as 'rosy' as they look.

The whole development with respect to NN and deep-learning is interesting, but to call it a new concept is nonsense, all these techniques are known for years, in the past it was just not feasible because there existed no suitable hardware to cope with these things in a reasonable amount of time.

Absolutely spot on. Mathew Lai of Giraffe used a 3-layer NN, better input layers than A0's etc, and got better evaluation than Stockfish while being only 10x slower. He was constrained by the fact that his engine had to perform well on single core CPU.

If he was given the luxury of massive hardware for accelerating the NN infernece, I am sure even a 5 year old can figure out the next step ... pile on more layers!

[hgm]

But the requirement that "an engine has to perform well on a single core x64 CPU" is completely non-sensical. That kind of logic leads to rejecting a ship as a viable means of transportation for a sea cruise because it doesn't perform very well on a Formula-I Grand-Prix racing track. So that eventually you will drown in your Ferrari.

[Daniel Shawul]

But the requirement that "an engine has to perform well on a single core x64 CPU" is completely non-sensical.

People choose how they want to compete and I am completely fine with it. There is equal-hardware tournament and non-equal ones for good reason.

I am only interested in algorithmic improvements and in that regard to be afforded the luxury of acceleration of a bulky eval is very unfair.

Though I originally thought there is a real algorithmic breakthrough ( convolutional nets and all), it turned out they absolutely did the same thing as DeepBlue to get the results that they got.

[frankp]

It is clear that lczero runs best on a graphics card. Why its merits can only be judged against its performance of a single cpu is bizarre. And after all it is a consumer grade card, not specialised hardware.

Yes, it will be limited by hardware - as everything ultimately is. But history suggests that hardware improves. Hope they can manage 256x20, which I believe A0 used.

Perhaps we should judge video games by only their ability to run a single cpu ... LOL

[hgm]

I am only interested in algorithmic improvements and in that regard to be afforded the luxury of acceleration of a bulky eval is very unfair.

But 'improvement' is a meaningless concept if you have no measure of quality. What would be an improvement in one metric, might be total degradation in another metric.

You see no algorithmic improvement in AlphaZero or LCZero, because you are using a completely flawed metric. Namely how well it performs on a single-core x64 CPU.

But no one cares about that. If you would use a more realistic metric, the neural network is a vast algorithmic improvement.

[frankp]

I understand the point, but is a commercial graphics card really 'unequal' hardware. Requiring every chess program to use only 64-bit integer manipulation (so to speak) because that what cpu excells at is extremely limiting.

We could equally require that AB searchers run on a graphic card.....

[Joost Buijs]

The AlphaZero paper only describes the differences with AlphaGo Zero, (and explicitly states that), so for everything that is not described there you will have to refer to the AlphaGo Zero paper.

To be honest I only read the paper very briefly, and I might have missed this.

An opening book is not Stockfish. This is like saying I am crippled when playing a Chess game because I am not allowed to have Kasparov make the first 25 moves for me. If that counts as 'crippled', then AlphaZero was just as much crippled. AlphaZero was not using EGTB (which do not provide measurable strength in a 100-game match anyway...). AlphaZero was using fixed time per move. That there are conditions under which Stockfish might perform better doesn't mean it is 'crippled'.

Stockfish is designed to use an opening-book and EGTB, and is optimized when using these, so yes it will be crippled when you let it play without.

The same holds for the enormous amount of threads they let it run on, usually engines using YBW are deteriorating above 16 threads or so and they used hyper-threading which effectively doubles SMP overhead.

You are accusing them of plain, premeditated fraud. For virtually no gain, jeopardizing the great fame they achieved through their Go results (which cannot possibly have been forged). Does that sound likely to you? And yes, it does happen that millionaire movie stars are caught shop-lifting at the super-market. But so rarely that it makes international news, and they are considered mentally ill. How likely is it that a large team of people would want to ruin their future that way? I never heard of a gang of millionaires going around at night to burgle houses, which would roughly be the equivalent. Suspecting this sounds like complete paranoia.

I'm not accusing them of anything, and I'm not paranoid either because I couldn't care less, I just said that things sometimes look better than they actually are.

Of course they achieved a wonderfull result with GO, but that doesn't mean the same technique is optimal for a very tactical game like chess. Maybe using a NN to guide an a-b search would give a better result in the end.

[jp]

The AlphaZero paper only describes the differences with AlphaGo Zero, (and explicitly states that), so for everything that is not described there you will have to refer to the AlphaGo Zero paper.

To be honest I only read the paper very briefly, and I might have missed this.

Please point to where exactly it explicitly states that everything not mentioned is exactly as in the AlphaGo Zero paper. If it's there, that's a helpful scrap of info.

Unfortunately, there's a lot of info missing from all of their papers.