vLLM Moves to LibTorch Stable ABI to Decouple PyTorch Upgrades

Introduction

vLLM makes extensive use of PyTorch’s LibTorch interfaces in its C++ and CUDA backend. Previously, a change in LibTorch’s binary interface could force vLLM to be rebuilt and relinked even when the source-level APIs used by the project had not changed. That coupling could delay support for new PyTorch releases and increase the cost of maintaining and distributing wheels.

The vLLM team has now described its migration to the LibTorch Stable ABI. Starting with v0.33.0, vLLM publishes CUDA wheels using the stable ABI. When other compatibility conditions are satisfied, users can replace the torch wheel and import it into an existing vLLM package instead of rebuilding vLLM for every compatible PyTorch update.

What the Stable ABI provides

An ABI defines how compiled code calls functions, lays out data, and resolves symbols. An unchanged API therefore does not guarantee that an already-compiled extension will continue to work. A stable ABI preserves a supported set of binary interfaces across compatible library versions.

PyTorch’s LibTorch Stable ABI has three layers: a stable C shim exposing selected LibTorch functionality, C++ wrappers that make the shim easier to use, and header-only utilities under torch::headeronly that do not link against libtorch.so. This is a deliberately limited interface rather than a promise that every LibTorch API is stable.

How vLLM approached the migration

• Inventory the dependency surface. The project reviewed its PyTorch calls, identified unsupported APIs, and planned the stable interfaces needed to close the gaps.
• Move unsupported mechanisms to Python. Stable SymInt and PyTorch autograd machinery were not available in the required stable form, so some meta functions and AutogradFunctions were moved from C++ to Python.
• Build a separate stable library. With more than 175 C++/CUDA source files and over 120 headers in the CUDA build, an all-at-once conversion would have been difficult to review. vLLM created a separate library and migrated kernels in smaller batches. The TORCH_TARGET_VERSION flag makes the build fail when unstable LibTorch headers are included.
• Register operators through the stable path. Stable operators use STABLE_TORCH_LIBRARY, with implementations wrapped by TORCH_BOX so C++ values can cross the ABI through generic StableIValue types. Code using other registration mechanisms may need to move to TORCH_LIBRARY first.
• Replace APIs file by file. Typical edits include changing torch::Tensor to torch::stable::Tensor, using stable wrappers for operations such as view, and replacing direct scalar references with header-only equivalents.
• Verify both behavior and symbols. vLLM combines unit and performance tests with line-by-line review of migration diffs. It also uses torch-abi-audit to inspect shared objects and classify stable versus unstable PyTorch symbols.

Tradeoffs and remaining constraints

The stable ABI only addresses LibTorch binary compatibility. Compatibility can still depend on Python APIs, GPU requirements such as the minimum supported CUDA version, and third-party packages. During the migration, is_pinned() did not have a stable equivalent. vLLM used the general-purpose torch_call_dispatcher to invoke the ATen operator through the stable ABI while preserving behavior; PyTorch 2.14 later added a stable C++ is_pinned() method.

The C shim can add host-side work for type conversion, operator lookup, and boxing. The reported cost is usually small and may disappear after CUDA Graph warmup, although eager execution with repeated launches of very fast kernels can make it more visible. vLLM reported no noticeable runtime regression during the migration.

Why it matters

The migration reduces one source of release coupling between an inference engine and PyTorch. Users may receive compatible PyTorch updates sooner, while maintainers and distributors can avoid some repeated rebuild and relink work. The process also gives PyTorch practical feedback about the interfaces frequently used by a large CUDA project.

This should not be read as a guarantee that any PyTorch version can be dropped into any vLLM environment. Instead, the Stable ABI is a narrower, testable compatibility layer. vLLM’s experience shows how a large CUDA project can use staged migration, build-time safeguards, symbol auditing, and runtime tests to make binary compatibility a manageable engineering target.

Source: vLLM Blog