What's the difference between TPU vs. GPU?

First, we had CPU and GPU. Then came AMD's Accelerated Processing Unit (APU, if you're nasty). And with the advent of the AI PC and smartphones, we've welcomed the NPU (Neural Processing Unit). But now, thanks to Google, we're getting yet another chip: the TPU. First internally deployed by Google back in 2015, the TPU, or Tensor Processing Unit, is the company's proprietary AI accelerator designed to handle all the complex mathematical calculations necessary for artificial intelligence and machine learning (ML) workloads. Designed for cloud-based applications, the TPU is used primarily for data centers. However, this term has recently popped up in the new Pixel 11 smartphone by way of its Google Tensor G6 chip, which the company says packs "50 percent more TPU compute."

So which is it? The answer is confusingly both. In the Pixel 11 series smartphones, the TPU is basically Google's fancy-schmancy way to say NPU. This version of a TPU handles camera and image processing as well as local AI tasks. But the data center TPU is decidedly different from a smartphone version, a traditional NPU and a GPU, although there are a few similarities. In order to help keep your chipset lingo straight, we're here to formally introduce you to the real TPU and clear up some of the misconceptions.

Great question. A TPU is a chip Google created to deal with cloud-based AI and machine learning tasks in the company's data centers. Known as an AI accelerator, a TPU is optimized to perform millions of mathematical calculations on the fly in AI models, something that NPUs and GPUs do as well.

We've been talking a lot about AI and ML, which for many of us will call into play an NPU. As it should, since we've been beaten over the head with them ad nauseum thanks to Apple, the various PC and smartphone makers and the chip manufacturers. If you're unfamiliar, NPUs are the chips you'll find in all your modern smartphones, Macs and PCs. They do all the heavy lifting for your generative and agentic AI tasks, whether it's erasing an ex from a photo or using ChatGPT to build out a three-day travel itinerary. For the Pixel 11 phones, the TPU replaces the NPU, with Google claiming "up to 3.5 times faster AI processing while using up to 3.5 times less energy."

GPUs are familiar to many of us, especially if you're a gamer or work with 3D models. Before AI, we wanted a powerful GPU to run graphically demanding games like Crysis 3 at the highest settings with blistering frame rates and buttery smooth rendering. Post-AI, GPUs still do that as well as handle AI training and crypto mining, hence why the best ones cost an arm, a leg and a firstborn. A jack-of-all-trades, a GPU can do many things well depending on how powerful it is.

The things that set these chipsets apart are largely use cases and scale. While TPUs are used for cloud-based AI running in Google data centers, NPUs, on-device TPUs and GPUs can be found in your smartphones and laptops.  

TPUs have the largest scale by far. Thanks to its specialized architecture, a TPU is employed in accelerating massive computations used in LLM (Large Language Models) and massive deep learning (DL) tasks. TPUs utilize a hardware layout called a systolic array that allows data to go from one unit to the next in a 2D grid of multipliers, which help eliminate potential bottlenecks. That means the calculation output becomes the input for the next unit without having to write back to memory, unlike GPUs, which are stuck juggling data between their compute units and their high-bandwidth memory.

When servicing big AI companies such as Anthropic and Midjourney, which serve billions of AI requests daily, relying on a GPU-based solution can be costly in terms of bottlenecks, latency and power efficiency. Since a TPU isn't cycling back and forth to read and write memory, it's a better fit for training large-scale neural networks and other massive workloads.

Living in our smartphones, IoT and laptops, NPUs and Google's on-device TPUs are relegated to the shallow end of the AI/ML pool. They're optimized for low-power tasks, which is why they're perfect for things like camera effects and real-time translation. A TPU can do an LLM, but it has to be a small and really well-defined local one.

"Better" is relative. Each AI accelerator is best in a specific use case. NPUs and on-device TPUs are better for handling the smaller mainstream AI tasks. Their low power consumption means they won't be too much of a draw on your smartphone, laptop or smartwatch. However, it's got enough kick to do real-time translation, blur your background during a call or help you remember that obscure show you swore you saw back in the '90s.

TPUs are the cost- and energy-efficient solution that helps run the AI/ML/DL show globally. Processing millions of mathematical calculations while training ML and DL models at massive scale means TPUs are purely for big data centers. With their innate flexibility, GPUs can take on both small- and large-scale AI workloads as well as run games, edit video and run simulations. If you want to run local LLMs, a beefy GPU is the way to go.