ChatGPT hardware

[Jouni]

285.000 CPU cores, 10.000 Nvidia V100 GPUs, 400 Gigabit/s Netzwerk

[smatovic]

285.000 CPU cores, 10.000 Nvidia V100 GPUs, 400 Gigabit/s Netzwerk

Nice rig, can it play chess?

I've heard training is done on a Nvidia DGX Pod with 256 nodes with each 8 A100 GPUs.

--
Srdja

[towforce]

285.000 CPU cores, 10.000 Nvidia V100 GPUs, 400 Gigabit/s Netzwerk

But how many users is that serving at the same time?

Thank you for the interesting information - but I can't help thinking about a simple webserver with some light back end processing (so it couldn't be served just from cache). If there was only one user, a single $1 CPU* would be plenty - but what if, at any one time, a million users were using it?

*to get a CPU for $1, you'd probably have to buy a thousand of them in reality

[Modern Times]

I'm sure that Magnum will make his contribution to this thread before too long...

[Jouni]

The hardware was used in training from 500 000 000 000 words. About serving: I got 99% of time this

"ChatGPT is at capacity right now."

More hardware needed .

[towforce]

The hardware was used in training from 500 000 000 000 words.

Thank you for clearing that up. I wonder why they used GPUs instead of TPUs?

About serving: I got 99% of time this

"ChatGPT is at capacity right now."

More hardware needed .

I've been saying this to people: a great free service is going to have extremely high demand - so yes - more hardware needed. But who's going to pay for that hardware when there seems to be no ROI to OpenAI for making this investment?

For me, Chat GPT is a strongly positive indicator for the future - but in the present, they'll either have to ration or charge.

[smatovic]

[...]
I've been saying this to people: a great free service is going to have extremely high demand - so yes - more hardware needed. But who's going to pay for that hardware when there seems to be no ROI to OpenAI for making this investment?

For me, Chat GPT is a strongly positive indicator for the future - but in the present, they'll either have to ration or charge.

AFAIK it runs in MS Azure (no Google TPUs) and there is already a pro access version present.

--
Srdja

[towforce]

Hmmmm... I think that TPUs (or equivalents) would be both cheaper to buy and cheaper to run than GPUs. I am guessing that right now there are no TPUs that can match the processing power of the top GPUs, and that would be the reason.

[smatovic]

Hmmmm... I think that TPUs (or equivalents) would be both cheaper to buy and cheaper to run than GPUs. I am guessing that right now there are no TPUs that can match the processing power of the top GPUs, and that would be the reason.

Since Nvidia Volta series there are TensorCores present on most Nvidia GPUs, mat-mul dedicated hardware, Nvidia Hopper arch has "Transformer Engines", for large language models, with mixed FP16 and FP8 precision, I can imagine Nvidia knows how to serve the AI market...

--
Srdja

[towforce]

Since Nvidia Volta series there are TensorCores present on most Nvidia GPUs, mat-mul dedicated hardware, Nvidia Hopper arch has "Transformer Engines", for large language models, with mixed FP16 and FP8 precision, I can imagine Nvidia knows how to serve the AI market...

Interesting. I would expect ML and video gaming to be different markets: there will be some people who want a GPU for both purposes, but I would expect them to be in a minority. So, overall, I would expect the market to reward specialisation - TPUs for the ML users and GPUs for video gamers.

Maybe some video games now benefit from tensor cores, and the ML market (especially for training) simply isn't big enough for TPUs to get the investment to keep up with where GPUs are at.