AlphaZero beats AlphaGo Zero, Stockfish, and Elmo

[Milos]

Y-scale in Fig.2 means Elo without added reference point (that is what that clumsy term google used actually means). It doesn't mean Elo difference between 2 programs, because if it was Elo difference you'd need a single curve, not 2 of them (it's pretty obvious if you just used your head a bit).

The text clearly states the Elo baseline (relative Elo = 0) for Fig. 2 is the respective traditional program with 40ms thinking time.

Your comment that has absolutely nothing to do with my actual claim (about obviously wrong scaling for SF in that figure) or Evgeniy's pretty absurd claim (that figure represents basically derivative of Elo performance) but it is instead stating totally irrelevant point regarding Elo baseline (Elo reference point = 0) tells basically that you didn't at all understand the argument, claims or anything that has been discussed but only felt the urge to jump in and demonstrate your "smartness" by "proving" me wrong.
Congratulations man, you got me.
Now you can go stare at the sun, count stars or do whatever "smart" thing you usually do.

[Adam Hair]

64 cores to 1 is not like 4096 to 64. Moreover even if it were, the dependency of strength of play vs TTD is also sublinear, and, certainly, is practically bounded. It is actually demonstrated in paper, page 7. Did you read it?

Figure 2 is completely bogus. If wish Google actually cited that reference that shows SFs Elo performance increase when going from 10s to 1min/move of under 20Elo.

So your are playing expert and even don't understand meaning of relative ELO?

Lol, I missed this pearl.
"Relative Elo" (btw. it's Elo not ELO), you just invented that, did you not?

Ok, so make a thought experiment. You may even actually perform it to get the point. Take two instances of Stockfish. Run a tournament between them, that'd emerge rating for both, which would be around 0 if you made everything right. Now increase the time control. Repeat. You'll still get zero, if you are still doing things right. Would that mean that there wasn't improvement? Do you get the concept now?

Y-scale in Fig.2 means Elo without added reference point (that is what that clumsy term google used actually means). It doesn't mean Elo difference between 2 programs, because if it was Elo difference you'd need a single curve, not 2 of them (it's pretty obvious if you just used your head a bit).
Beside not understanding much, you really don't seem too bright.
Sorry, but I don't want to waste any more time with you, it's pointless.

If you are going to be exceedingly arrogant, at least make sure you do not make mistakes. The 2 curves represent the Elo difference of SF 8 and of AlphaZero from SF 8 at 40ms/move as the thinking time varies.

[Milos]

You claimed TPU is 180 TOPS, while it is 180 TFLOPS per pod of four TPU, thus your evaluation 4 times higher than it should be.

From what I can find, Google seems to be calling a motherboard containing 4 ASICs a Cloud TPU, and a TPU pod is 64 Cloud TPUs.

The guy is just a cocky kiddo with a keyboard faster than his brain. It's a lost cause really to argue anything he says.

Just a quote from https://ai.google/tools/cloud-tpus/:

Each device delivers up to 180 teraflops of floating-point performance, and these new TPUs are designed to be connected into even larger systems. A 64-TPU pod can apply up to 11.5 petaflops of computation to a single ML training task.

[Milos]

If you are going to be exceedingly arrogant, at least make sure you do not make mistakes. The 2 curves represent the Elo difference of SF 8 and of AlphaZero from SF 8 at 40ms/move as the thinking time varies.

If you are nitpicking they represent Elo performance of Alpha0 and SF vs thinking time with Elo baseline (Elo = 0) taken as Elo of SF8 at 40ms/move which is pretty much equivalent to what I've said up in the thread. So writing something just to write is pretty pointless.
Normally you don't have such a knee-jerk reaction, but hey everyone can have a bad day .

P.S. Instead try being constructive. Have you seen the paper, I guess you did? How you explain this Fig. 2 performance of SF gaining 20Elo going from 10s to 1min thinking time? Pretty sure if they are ever gonna publish it in Nature they'd have to change it.

[Adam Hair]

Y-scale in Fig.2 means Elo without added reference point (that is what that clumsy term google used actually means). It doesn't mean Elo difference between 2 programs, because if it was Elo difference you'd need a single curve, not 2 of them (it's pretty obvious if you just used your head a bit).

The text clearly states the Elo baseline (relative Elo = 0) for Fig. 2 is the respective traditional program with 40ms thinking time.

Your comment that has absolutely nothing to do with my actual claim (about obviously wrong scaling for SF in that figure) or Evgeniy's pretty absurd claim (that figure represents basically derivative of Elo performance) but it is instead stating totally irrelevant point regarding Elo baseline (Elo reference point = 0) tells basically that you didn't at all understand the argument, claims or anything that has been discussed but only felt the urge to jump in and demonstrate your "smartness" by "proving" me wrong.
Congratulations man, you got me.
Now you can go stare at the sun, count stars or do whatever "smart" thing you usually do.

He commented on your nonsensical statement of "Elo without added reference point", which is obviously not what the y axis represents.

As far as your actual claim, the scaling could be due to the test conditions, specifically the lack of opening positions. I am planning on checking this out to see if I can replicate the scaling.

[Milos]

He commented on your nonsensical statement of "Elo without added reference point", which is obviously not what the y axis represents.

So reference point of Y-axis of 0 (I am not talking about X-axis which is 40ms) is not equivalent to "without reference point"???
So you have solution of integral and it is something + constant, and I say solution is something without constant and you come and say no solution is something with constant =0.
And you are not nitpicking, gee...
I don't mind you being a jerk from time to time, but you can also be quite a hypocrite when you criticize ppl for being arrogant, and then you behave in exactly same manner...

[Milos]

As far as your actual claim, the scaling could be due to the test conditions, specifically the lack of opening positions. I am planning on checking this out to see if I can replicate the scaling.

So you plan to play SF8 against itself always from root position and with fixed time per move? Well I suggest you play on a single core. Maybe you discover something new like that SF's scaling function is a step function, maybe you also rediscover the wheel in the process .

[EvgeniyZh]

You claimed TPU is 180 TOPS, while it is 180 TFLOPS per pod of four TPU, thus your evaluation 4 times higher than it should be.

From what I can find, Google seems to be calling a motherboard containing 4 ASICs a Cloud TPU, and a TPU pod is 64 Cloud TPUs.

Yeah, well, per device with 4 TPUs.

BTW, new gen NVIDIA GPUs are claimed to have 100+ FP16 TFLOPS, which would made AlphaZero on consumer PC reality. 2xTitan V will match 4 TPUs, however, that needs some test of course.

[EvgeniyZh]

You claimed TPU is 180 TOPS, while it is 180 TFLOPS per pod of four TPU, thus your evaluation 4 times higher than it should be.

From what I can find, Google seems to be calling a motherboard containing 4 ASICs a Cloud TPU, and a TPU pod is 64 Cloud TPUs.

The guy is just a cocky kiddo with a keyboard faster than his brain. It's a lost cause really to argue anything he says.

Just a quote from https://ai.google/tools/cloud-tpus/:

Each device delivers up to 180 teraflops of floating-point performance, and these new TPUs are designed to be connected into even larger systems. A 64-TPU pod can apply up to 11.5 petaflops of computation to a single ML training task.

You butthurt is understandable, after all, everything you've done was beaten with 4 hours of computer work, and proofed you're useless. So the only thing you have left is to lie to everyone so that they have erroneous feeling you worth something. It's ok, after all, if persuading some forum guys you are not stupid makes you feel better, go for it.

Anyone with a bit of brain can understand (by looking on picture, if you can't read) that 180 TFLOPS is per device contatining 4 TPUs, which is called "Cloud TPU" as noted in post you replied to. Fun fact: out of all sources you choose the one that can be misread easily.

Let me predict you answer: more insulting, no proofs, nitpicking, ignoring the truth

[Lyudmil Tsvetkov]

4 hours my ass (pardon my french).

Far fewer transistors and joules were used training AlphaZero than have been used training Stockfish. You can soon rent those TPUs on Google's cloud, or apply for free access now, so stop complaining. Furthermore it's an experimental project in early days and performance is obviously not optimal, so all the 'but-but-but 30 Elo because they used SF 8 instead of SF 8.00194' sounds really dumb.

Days of alpha-beta engines have come to an abrupt end.

-Carl

Sorry, that is pretty childish rent.
Google is obviously comparing apples and oranges and again doing marketing stunt and ppl are falling for it.
Days of Alpha0 on normal hardware are years away. But keep on dreaming, no one can take that from you.

P.S. Just as a small comparison. leelazero open source project trying to replicate alpha0 in Go, took 1 month to get the same games as AG0 got in 3 hours, that with constant 1000 volunteers.
For chess it would take even more.

Training AlphaZero would take tons of time. Just like creating SF from 0. However, running it took 4 TPU, which is comparable to whats available to (rich) consumers - you can get 6-8 NVIDIA V100 which would get you similar performance.

To me this is the most informative post in the whole thread, assuming it is accurate (I know nothing about TPUs). The only reasonable comparison I can think of between the AlphaZero hardware and the Stockfish hardware is cost of equivalent machines. It doesn't matter to me how much hardware was used to reach the current level of strength for both engines, just whether the playing conditions were fair. You seem to be implying that comparable hardware to the 4 TPUs would cost no more (maybe much less?) than the sixty-four core machine used by SF. Is this correct? I'm asking to learn, not making a claim myself either way.

The other conditions were of course not "fair", but reasonable given that AlphaZero only trained for a few hours. I suppose if Stockfish used a good book, was allowed to use its time management as if the time limit were pure increment, and used the latest dev. version, the match would have been much closer, but probably (judging by the infinite win to loss ratio and the actual games) SF would have still lost. The games were amazing.

Bottom line, assuming the comparable cost claim is accurate: If Google wants to optimize the software for a few weeks and sell it, rent it, or give it away, we have a revolution in computer chess. But my guess is that they won't do this, in which case the revolution may be delayed a couple years or so.

Larry, what kind of revolution, this is 30/1 hardware advantage.
Alpha is currently at 2850 level.

Based on the estimated $60k price for equivalent hardware vs. maybe $20k for the 64 core SF machine (my guess) it would be 3 to 1, not 30. The actual hardware used by Alpha would be useless for SF, so you can't compare the hardware any other way than by price, I think. It sounds like the cost of the type of hardware needed for Alpha is expected to plummet while the cost of normal CPUs just trends slightly lower. We already see the same thing in GO. Leela (top GO engine for pc, and free like SF) plays a stone or so stronger with even a cheap GPU than without one. So while Alpha might only play at 2850 level on your laptop, it might be super-strong in a year or so on something many people could afford. But if Google doesn't release it, that won't happen.

Calculating performance by price? Seems like a very unscientific approach.
Maybe 60k is the price for just one TPU, while it used 4.
Why do you leave SMP inefficiencies aside?
What about the much bigger memory available to Alpha?
What about the book it had based on human games?
What about the fixed TC?
And the hash?

It is not a single thing, but 10 things that favour Alpha, so no one can convince me the real advantage was lower than 30/1.

No, they won't release it, because they will not make much more progress in the future.

And all those, who will be able to buy such hardware, will certainly not be chess players or even programmers, so how could that benefit chess or programming in any way?