GPU rumors 2020

[smatovic]

Intel Xe GPU update..

Xe-LP (low power) will debut as integrated graphics with up to 768 compute cores
around September in TigerLake CPU and as DG1 discrete graphics based on SuperFin
10 nm.

https://www.anandtech.com/show/15973/th ... the-bottom

Xe-HPG (high-performance gaming) is announced for 2021, build in an external fab.

https://www.anandtech.com/show/15974/in ... dparty-fab

TensorCores equivalent is named XMS, Xe Matrix eXtensions, and is
planed for future products, not sure when.

https://www.anandtech.com/show/15973/th ... e-bottom/4

Intel Xe architecture will support FP16 and INT8 operations with higher throughput.

--
Srdja

[smatovic]

Let me close this thread with an outlook what we have for 2021 in pipe...

Nvidia Hopper:

Next Nvidia architecture will be Hopper, switch to multi-chip-modules design,
maybe TSMC 5nm, maybe 2021/2022.

Intel Xe GPU:

Intel Xe-HPG, high-performance-gaming, is planed for 2021 in external fab,
unknown if XMX, Xe Matrix eXtensions, (like TensorCores?) will be onboard.

Intel AMX:

Advanced Matrix eXtensions for CPUs are planed for Xeon line in 2021, could be
used to boost NNs in NNUE and alike.

AMD RDNA/CDNA:

Not sure what AMD has in pipe, RDNA 2 is present in recent PlayStation/Xbox,
should be released as discrete GPU this year, unknown if TensorCores for
boosting CNNs will be present.

Apple:

Switch to ARM based CPUs in their Mac line, unknown what kind of GPU they will
use, some kind of neural network accelerator might be present, some kind of
CPU vector unit will be present like NEO (128 bit) or SVE (up to 2048 bit) for
boosting NNUE.

Aurora/Frontier:

Exa-FLOP systems Aurora (Intel) and Frontier (AMD) are planed for 2021, will use
'gpu-like' accelerators, maybe already with unified memory architecture between
CPU and GPU.

--
Srdja

[smatovic]

Alrighty, this is a beast, AMD Arcturus MI100 with CDNA architecture for HPC:



https://www.nextplatform.com/2020/11/16 ... elerators/

92.3 TFLOPS FP16 and 184.6 TFLOPS via its new Matrix Cores.

Yes, sad, 'pricey' and for HPC, but this move makes it clear that the path for Gaming and HPC GPUs differs.

AMD has now RDNA gaming arch w/o and CDNA arch with Matrix Cores, Intel plans XMX matrix cores at least for its Xe-HPC series, so I will predict that Nvidia will join and drop TensorCores from its next RTX gamer brand and the best shot for Lc0 will be the RTX 2080 TI for a long time...maybe.

--
Srdja

[Dann Corbit]

I wonder how the AMD MI100 compares to the Nvidia A100:
https://nvidianews.nvidia.com/news/nvid ... blZhIn0%3D

[smatovic]

I wonder how the AMD MI100 compares to the Nvidia A100:
https://nvidianews.nvidia.com/news/nvid ... blZhIn0%3D

https://en.wikipedia.org/wiki/Ampere_(m ... d_DGX_A100

Both GPUs are based on TSMC 7nm process, A100 with 400W MI100 with 300W TDP, hence I guess meanwhile the TDP is the limiting factor in overall throughput of HPC GPGPUs and AMD has here some cores and frequency in spare. As for Lc0, might need a new backend type to make use of the new Matrix Cores, not sure about that.

--
Srdja

[Milos]

Alrighty, this is a beast, AMD Arcturus MI100 with CDNA architecture for HPC:



https://www.nextplatform.com/2020/11/16 ... elerators/

92.3 TFLOPS FP16 and 184.6 TFLOPS via its new Matrix Cores.

Yes, sad, 'pricey' and for HPC, but this move makes it clear that the path for Gaming and HPC GPUs differs.

AMD has now RDNA gaming arch w/o and CDNA arch with Matrix Cores, Intel plans XMX matrix cores at least for its Xe-HPC series, so I will predict that Nvidia will join and drop TensorCores from its next RTX gamer brand and the best shot for Lc0 will be the RTX 2080 TI for a long time...maybe.

--
Srdja

Beast without framework is a joke. AMD's pathetic attempts to go into ML market are pretty pointless. Well good luck to them with OpenCl.
Btw. in 7nm having best performance in BFLOAT16 of 0.5TOPS/W is pathetic. This coming from someone actually designing HPC AI hardware.

[smatovic]

...
Beast without framework is a joke. AMD's pathetic attempts to go into ML market are pretty pointless. Well good luck to them with OpenCl.
Btw. in 7nm having best performance in BFLOAT16 of 0.5TOPS/W is pathetic. This coming from someone actually designing HPC AI hardware.

Nvidia invested early in CUDA and framework ecosphere, AMD is a bit scattered here IMO. But in HPC it also about the closed/open source paradigma. There is a point in investing tens or hundreds of millions dollars in hardware and having the software ecosphere as open source available...

--
Srdja

[Milos]

...
Beast without framework is a joke. AMD's pathetic attempts to go into ML market are pretty pointless. Well good luck to them with OpenCl.
Btw. in 7nm having best performance in BFLOAT16 of 0.5TOPS/W is pathetic. This coming from someone actually designing HPC AI hardware.

Nvidia invested early in CUDA and framework ecosphere, AMD is a bit scattered here IMO. But in HPC it also about the closed/open source paradigma. There is a point in investing tens or hundreds of millions dollars in hardware and having the software ecosphere as open source available...

--
Srdja

Open source won't have any edge here no matter what. You need low level access to hardware for that, and open source is not gonna get it.
I predict that Intel will do MKL version for their AI hardware like Nvidia did with cuBlas.
AMD is simply not good enough in writing software and frameworks. They are hardware company to the bone. That might change in the future, but it is nowhere near on the horizon now.

[smatovic]

...
Beast without framework is a joke. AMD's pathetic attempts to go into ML market are pretty pointless. Well good luck to them with OpenCl.
Btw. in 7nm having best performance in BFLOAT16 of 0.5TOPS/W is pathetic. This coming from someone actually designing HPC AI hardware.

Nvidia invested early in CUDA and framework ecosphere, AMD is a bit scattered here IMO. But in HPC it also about the closed/open source paradigma. There is a point in investing tens or hundreds of millions dollars in hardware and having the software ecosphere as open source available...

--
Srdja

Open source won't have any edge here no matter what. You need low level access to hardware for that, and open source is not gonna get it.
I predict that Intel will do MKL version for their AI hardware like Nvidia did with cuBlas.
AMD is simply not good enough in writing software and frameworks. They are hardware company to the bone. That might change in the future, but it is nowhere near on the horizon now.

Hmm, I think I agree. AMD did their Zen thing right, maybe now resources for GPGPU frameworks? Remains open if Nvidia/AMD/Intel will successfully push CUDA/ROC/oneAPI or if they can agree to an open standard again...maybe I am a dreamer

--
Srdja