How vLLM Adapts LLM Serving to Tenstorrent’s Mesh Architecture

Introduction

vLLM has presented a plugin that brings Tenstorrent accelerators into its serving stack. Once installed, the plugin detects Tenstorrent hardware whenever TT-Metal’s ttnn runtime is importable and registers it as a vLLM platform. For application developers, the serving contract remains familiar: the OpenAI-compatible API, request formats, and client code do not need to change.

The important part is not simply adding another backend. Tenstorrent systems are organized as meshes of cores and chips, with cross-chip movement handled by a compiled program and an on-fabric network. That differs materially from the GPU-oriented assumption that a host launches work and issues collectives across runtime ranks. The plugin therefore adapts several layers of vLLM’s serving behavior without putting Tenstorrent-specific logic into the core project.

Key design points

• An out-of-tree integration. The plugin registers a platform, worker, scheduler, and model architectures through existing vLLM extension points. Tenstorrent options are passed through the generic additional-config namespace rather than new core CLI flags.
• Mesh execution instead of rank-based parallelism. A model is compiled and traced for the complete target mesh. MESH_DEVICE selects the device configuration, while traditional -tp and -pp options are rejected rather than being presented as if they had their usual meaning. The model implementation determines the parallelization that fits a given mesh.
• Phase-constrained scheduling. Each step is either prefill-only, decode-only, or empty. Prefill and decode are not mixed in the same batch. Long prompts can still be split across multiple prefill steps, with decode steps interleaved so active requests continue progressing.
• A preference for stable shapes. Device execution relies heavily on replaying traced programs for fixed batch shapes. Homogeneous, shape-stable batches are therefore a better match than highly heterogeneous ones.
• Sampling can remain on the device. Because the mesh program may extend through sampling, some modes return the selected token without sending full logits back to the host, while a host-side fallback remains available.
• Broader model coverage. The listed implementations span Llama, Qwen, Mistral, Gemma, DeepSeek, and GPT-OSS families, with support for selected vision and multimodal architectures. Additional model bundles can be registered from an external directory without editing the plugin source.

Why it matters

The project is a useful test of whether a serving framework’s plugin boundary is genuinely hardware-agnostic. Rather than pretending that a Tenstorrent mesh behaves like a collection of GPU processes, the integration keeps mesh compilation, scheduling constraints, and sampling behavior in the plugin and TT-Metal model implementations. That preserves vLLM’s API and ecosystem while limiting changes to upstream core.

There are trade-offs. Operators cannot simply transfer GPU tensor-parallel settings to this path, and batching policies must account for separate prefill and decode phases. Hardware vendors, however, gain a more maintainable route into a fast-moving serving framework: they can track vLLM through an extension interface instead of carrying a long-lived fork.

The broader lesson is that an inference framework’s extensibility is not measured only by whether it can discover new hardware. It is measured by whether it can express a different execution model without forcing every backend into the same GPU-shaped abstraction.

Source: vLLM Blog