Google's bfloat for neural networks

[smatovic]

Hehe, funny, Google started to use their own bfloat datatype in TPU gen 2 and gen 3 for neural networks,

https://en.wikipedia.org/wiki/Bfloat16_ ... int_format
https://www.nextplatform.com/2018/05/10 ... processor/

and now Intel starts to implement it in their hardware. That's when you know you are a big player

https://venturebeat.com/2018/05/23/inte ... -training/

Wonder if Nvidia or AMD will join.

--
Srdja

[mar]

I misread as "Google's bloat...", thought that Google open sourced yet another masterpiece

So this bfloat16 is basically float where you throw away 16 bits worth of mantissa.
Packing/unpacking from 32-bit float should be trivial, so probably clever, but hey only 7 bits of mantissa, is it really enough?

[smatovic]

I misread as "Google's bloat...", thought that Google open sourced yet another masterpiece

So this bfloat16 is basically float where you throw away 16 bits worth of mantissa.
Packing/unpacking from 32-bit float should be trivial, so probably clever, but hey only 7 bits of mantissa, is it really enough?

Dunno

https://www.hpcwire.com/2019/04/15/bsc- ... -training/

"As training progresses and it hones the value of the weights, then greater precision becomes important in order to optimize the solution."

“We believe dynamic numerical precision approaches offer the best benefit to training and inferencing,”

--
Srdja