Milos wrote:It actually is, instead of 4TPUs required to run Alpha0 so far, on x64 hardware one would need around 2000 Haswell cores to achieve the same speed of NN
Details about Google's TPU are available here.
https://drive.google.com/file/d/0Bx4haf ... xtcEk/view
For NN inference, 1 TPU is around 15-30 times faster than an Haswell (with multiple cores, so your figure of 2000x faster for 4 TPUs vs 1 core is right), but other comparable properties favor the CPU:
TPU vs CPU
Die size: TPU is < half size of an Haswell
Frequency: 700 Mhz vs 2300
Power: 40W vs 145W
Memory Bandwitdth: 51 vs 34
The TPU is much more efficient than a CPU (for NN inference), but it does not eat more power, occupies more size, etc.
So using 4 TPU vs 64 Haswell cores, is fair I think.