A question about speed difference between gpu and cpu

[Uri Blass]

I understand that gpu is faster for LC0 relative to cpu but other engines cannot use the gpu in a way that it is faster than cpu.

Are there simple mathematical problem to solve when cpu is faster than gpu and simple mathematical problems to solve when gpu is faster than cpu

I would like to see them including speed difference.

I understood that LC0 multiply matrixes and I would like to know if multiplying matrixes can be done faster by the gpu and what is the speed difference.

I would like to see 2 type of problems when computers can solve everyone of them with TCEC hardware in 0.01 second.
When for one of them the optimal way to do it in 0.01 seconds is going to be to use CPU and for the second problem the optimal way is going to be to use GPU

I would like to know what is the harder problem out of them for humans who are not allowed to use computer help during the solution and and can use only pen and paper.
harder means that humans need more time to solve

[jorose]

I am no expert on the topic, but hopefully my understanding is not completely off and I can give you somewhat of an intuition.

So first one needs to realize we are not just talking about a bit of a difference in theoretical compute power, but an order of magnitude. To elaborate, the 2x1080TI that they gave Leela in TCEC has a total of 7'160 cores, each running at the respectable frequency of 1.6 GHz. A quick google search tells me the core i7-8700k offers 61 GFlops, the 1080TI offers 11.4TFlops (11'400 GFlops) and the V100, which TCEC staff was asking people if they wanted, offers 112 TFlops (112'000 GFlops) for Deep Learning applications; the equivalent of 1'800 core i7-8700k s

Unfortunately while the speed difference between an individual GPU core and a CPU core is not huge, there are a decent set of features missing from GPU cores that are in every modern CPU. Eg virtual memory and interrupts. I am not certain which features would need to be added to GPUs in order for them to be used as general use processors or even just reasonable processors for chess engines.

Despite this, GPUs are theoretically Turing complete, so if we can feed big block of highly parallelizable computations onto it than there is a huge potential for massive speedups. Neural net computations are generally highly parallelizeable. Each layer in the current Leela test net is has 256 filters. What this means is that there are 8x8x256 larger computations (each consisting of 3x3x256 multiplications and additions) that can essentially be calculated independently of one another and can (and I afaik is, but don't quote me on that) be implemented as a matrix-matrix multiplication.

To make matters more extreme, specialized compute power for neural nets is expected to increase significantly faster than Moore's Law. Iirc in the recent OpenAI Five benchmark, one of the OpenAI members mentioned that there is currently a doubling going on every 3.2 months.

[Sesse]

The difference between a CPU and a GPU core is huge. Each core in a GPU is about equivalent to a 70–80 MHz CPU core, give or take. (It's clocked at 1.6 GHz, sure, but each instruction takes a lot more cycles.) But you have tons of them, and they are widely SIMDed.

Quick rule of thumb: If you could efficiently use a 1000-core CPU, you're likely to be able to benefit from a GPU.

[jorose]

The difference between a CPU and a GPU core is huge. Each core in a GPU is about equivalent to a 70–80 MHz CPU core, give or take. (It's clocked at 1.6 GHz, sure, but each instruction takes a lot more cycles.) But you have tons of them, and they are widely SIMDed.

Quick rule of thumb: If you could efficiently use a 1000-core CPU, you're likely to be able to benefit from a GPU.

Could you explain why the Flop numbers are so extreme? I might have misunderstood something, but for me a Flop is defined as a floating point operation. which Ill admit is vague, but should be the same for both?

[Sesse]

I don't understand the question. Why should a CPU and a GPU be able to process the same amount of floating-point operations (typically 32-bit multiply-adds) per second?

[odomobo]

This talk is, by far, the best explanation I've seen of how GPUs operate vs CPUs: https://youtu.be/KHa-OSrZPGo

After watching it, you should have a strong understanding of what makes them fast, and what is/isn't practical to perform on a GPU.

Each core in a GPU is about equivalent to a 70–80 MHz CPU core, give or take.

Even if that's sometimes true, it's a pretty misleading statement. In reality, the nature of the GPU's architecture drives its efficiency for a particular task, and they are able to achieve full utilization for some tasks (without requiring more instructions than it would require on a CPU).

[Sesse]

I don't see what's misleading about it. It captures the point of a GPU: The individual “cores” are slow, but you have tons of them. So you need a massively parallel problem to get good use out of it.

Of course, actual GPU design is a lot more complicated, but CPUs are super-complicated, too.

[oreopoulos]

The difference is in the architecture.

You use a single command (for example add) to multiple data.

So you have 2 vectors V1 and V2. You store them in "register vectors" and adding is just a single command. It does not require any loop.

You can also multiply subtract... etc.

But complex logic is not efficient in such vector architecture.

So anything that has to do with Linear Algebra is very fast (for example games which is just plain Analytic Geometry)

For that specific problem, the speed up can be x100 or more depending on the card and CPU.

[Sesse]

You are aware that CPUs also have SIMD instruction sets, right?

[oreopoulos]

Of course. But they are limited in register size.