TPUs Run Kimi 57% Faster: Inference Software Is the Real Differentiator

Introduction

Google’s TPUs may be gaining ground against Nvidia GPUs in large-model inference, but the deciding factor is not necessarily peak hardware throughput. Inferact reports that 16 TPU v7 Ironwood chips running Kimi K3 achieved 709 tokens per second, compared with 452 tokens per second on 16 GB200 GPUs. That is a claimed 57% lead. The more important story, however, is the software stack built around the hardware.

What the benchmark shows

• Both systems used the same Kimi K3 model and the vLLM inference engine. The main differences were the accelerators and their low-level kernels.
• The TPU setup used DSpark speculative decoding, in which a smaller model proposes tokens and the larger model verifies them in batches. The reported average acceptance length was six.
• With speculative decoding disabled and batch size set to one, TPU v7 reached 249 tokens per second versus 127 for GB200. At batch size eight, the figures were 865 and 636 respectively.
• In a separate Qwen 3.8 27B test, four TPUs reached 1,515 tokens per second, while four GB200 GPUs reached 695.
• Inferact reports no quality loss for its Kimi setup: GPQA-Diamond scored 94.4% and GSM8K 97.2%, matching its GPU results.

These are vendor-reported benchmarks. They demonstrate the potential of the implementation, but do not establish a universal performance ranking across models and serving conditions.

Why the Megakernel matters

Autoregressive decoding is often limited by moving data rather than by arithmetic. Conventional serving breaks inference into many small kernels. Each handoff introduces launch and scheduling gaps, leaving memory bandwidth underused. Inferact’s Megakernel instead places the multi-layer execution path of Kimi K3 into a single Pallas program.

This design allows computation in one layer to overlap with prefetching weights for the next. TPU on-chip VMEM is explicitly managed by software, giving engineers more control over what is loaded, retained, and released. Inferact says it bypassed XLA’s normal layer-level optimization path and manually coordinated data movement across the model. Compilation reportedly fell from more than 30 minutes to under 90 seconds.

Implications and limits

The result highlights a broader shift in inference competition: effective dataflow and memory scheduling can matter as much as advertised compute specifications. For TPUs, wider support in open-source serving frameworks could be as important as new silicon. Nvidia’s CUDA Graphs and PDL point toward the same general goal of reducing execution overhead.

There are clear limitations. The current Megakernel is tailored to Kimi K3 and still requires adaptation for other architectures. Its long-term significance will depend on whether it can support more models, integrate smoothly with vLLM, and retain its advantage under real-world latency, cost, and concurrency requirements.

Source: QbitAI