What Nvidia's first Groq 3 LPU benchmarks do and don't tell us about its $20B gamble

Nvidia’s $20 billion bet on Groq’s LPU tech sure looks like it was a good one. On Monday, the GPU giant offered the first glimpse of just how big a speedup its Groq 3-based LPX racks will provide.

In an independent benchmark conducted by Artificial Analysis, Nvidia’s LPX rack systems managed to churn out 3,400 tokens a second (tok/s) with a 100,000-token input sequence in Google’s Gemma 4 31B model.

According to Nvidia, this makes it 4x faster than the nearest alternative platform, which going off Artificial Analysis’ leaderboard would be a direct dig at Cerebras, which managed a still impressive 882 tok/s under the same conditions.

Acquihired by Nvidia in late December, Groq has LPUs that feature an SRAM-heavy dataflow architecture designed specifically for high-performance inference serving. Unlike traditional datacenter GPUs, which rely on high-speed DRAM memory tech like GDDR7 and HBM4, Groq’s chips rely entirely on a large pool of on-die SRAM orders of magnitude faster than even the best HBM stacks (around 2.75 TB/s each) available today. And when it comes to inference, memory bandwidth is the bottleneck to beat.

The third generation of the chips launched as part of Nvidia’s broader Vera Rubin platform earlier this year boasted 150 TB/s of memory bandwidth. The catch, of course, is that SRAM chews up a ton of die area, which means you can’t pack all that much into a reticle-sized part. Where Nvidia’s top-specced Rubin GPU boasts 288 GB of onboard memory, each Groq 3 LPU has just 500 MB — 576x less.

That’s far too little memory to run Gemma 4 31B on a single LPU, so Nvidia’s architecture uses Ethernet to distribute models across multiple accelerators. Each LPX rack can be equipped with up to 256 LPUs for 128 GB of high bandwidth SRAM. For large models, multiple LPX racks can be ganged together.

So why would anyone need to run Gemma 4 31B at 3,400 tokens a second? You probably don’t, but the AI code assistant or agent you're using it with might. The faster you can generate tokens, the longer models can reason, the more turns the agent can take, and the more information it can process or actions it can take in the same window of time.

The big bet here is that, in the agentic age, faster inference servers make for smarter agents and are therefore worth a premium. And the combination of Nvidia GPUs and Groq 3 LPUs certainly seems to have caught Nvidia’s customers’ attention. On Monday, the GPU giant revealed that Netherlands-based neocloud Nebius would be among the first to field the combined systems in its datacenters.

Just how impressive is Nvidia’s LPX system really?

While 3,400 tok/s is impressive at face value, Gemma 4 31B is arguably a best case scenario for the hardware, and it remains to be seen how gracefully the architecture will scale to larger, more complex MoE models.

At 31 billion parameters, the model fits neatly into a single LPX rack regardless of what data type Nvidia is using to store the weights. Nvidia is running the model at FP8, which means it needs a little over 31 GB or just under 64 LPUs of SRAM capacity.

We’ve reached out to Nvidia for comment on how exactly it's distributing the model across those chips, but didn’t get a clear answer. Our best guess is plain old pipeline parallelism with possibly a dash of data parallelism to improve performance at higher concurrency levels.

Pipeline parallelism, if you’re not familiar, distributes the model across the accelerators and then processes decode operations in one great big pipeline. Since there are enough LPUs, by our approximation, to hold four copies of the model, data parallelism would effectively duplicate this pipeline and provide additional independent slots for higher concurrency.

If you’re thinking that 31B isn’t very large, you’re right. The model is small enough to run on a high-end consumer graphics card like an RTX 3090/4090 — at 4-bit precision of course.

While it may be a small model, we wouldn’t write off Nvidia’s test results just yet. The model itself is relatively small, but it’s a dense model, which means all 31 billion parameters are activated each time a token is generated. Thirty-one billion also happens to be fairly close to the number of active parameters used by much larger MoE models like DeepSeek’s 671 billion parameter V3, which has 37 billion active parameters.

It should be noted, however, that because MoE models use different parameters for each token generated, they suffer from performance overheads that you don’t see in dense models.

The other problem worth noting is that to serve a model like DeepSeek V3 on an LPX system would require 1,342 accelerators or just over 5 LPX racks. That’s a lot of accelerators to wrangle.

Taking LPX to the next level

The performance figures shared by Nvidia today are for a model running entirely on LPUs. However, Nvidia’s biggest performance gains will come from combining its GPUs with the dataflow accelerators in a heterogeneous inference architecture.

We discussed this approach in more detail earlier this year, but, in a nutshell, what Nvidia is doing is breaking the inference workload into two distinct phases. The compute heavy prefill phase, during which prompts are processed and the key-value caches used to track model state are generated, is done on its GPUs, while the memory-bandwidth intensive decode phase is offloaded to the Groq 3 LPUs.

The result is a massive speedup in inference performance beyond what GPUs alone could deliver. The chart below summarizes how Nvidia’s GPUs and LPUs fit together nicely.

Rubin’s high compute density makes it ideal for high volume, bulk inference where the goal is reasonable interactivity (tok/s per user) for a large number of concurrent users, while LPUs are best suited for maximum interactivity. By combining the two, Nvidia has managed to lift the Pareto curve up and to the right, maximizing throughput without compromising on interactivity.

The 3,400 tok/s recorded by Artificial Analysis reflects performance on the far right of the chart, but by combining Groq’s LPUs with its GPUs, Nvidia expects it can achieve a decent fraction of that performance across a large number of concurrent users.

Nvidia’s figures may not age well

Along with the LPUs’ potential scaling challenges, it’s also worth discussing Nvidia’s comparison against Cerebras’ chips. Nvidia claims its systems are 4x faster than Cerebras' parts, which is true. Artificial Analysis testing at a 100,000 input sequence (prompt) does corroborate this. What is left out of that conversation is the number of chips required.

We don’t know the specifics of Cerebras' config, but it’s not hard to suss out. Cerebras usually serves its models at mixed precision (eight and 16 bits) with 16-bit activations, which means the entire Gemma 4 31B model should fit into one, at most two 44 GB CS-3 accelerators, whereas Nvidia needs at least 64 chips.

Cerebras' performance figures also don’t reflect its next-generation accelerators announced last week.

The WSE-3T-based CS-4 accelerator doubles the compute, I/O bandwidth, fabric speed, and memory bandwidth of its existing chips, and also triples the number of accelerators per rack. The new systems are expected to launch on Cerebras' inference cloud later this year alongside a heterogeneous GPU+WSE-based system similar to Nvidia's NVL72+LPX config we discussed earlier.

The tie up, announced at AMD’s Advancing AI event last month, will see the House of Zen combine its Helios GPU rack systems with Cerebras dinner-plate accelerators. Cerebras is working with AWS on a similar config as well.

As such, Nvidia’s Gemma 4 31B performance comparison may not age well. ®