vLLM on Arm CPUs: From Basic Enablement to Faster LLM Serving

Introduction

CPU-based LLM serving remains an important deployment option because CPUs are widely available, relatively simple to operate, and often cheaper to deploy than specialized accelerators. As Arm Neoverse servers become more common in cloud and enterprise environments, the quality of the Arm CPU path in open-source serving frameworks such as vLLM matters more. A recent vLLM blog post outlines months of upstream work with the vLLM, PyTorch, oneDNN, and KleidiAI communities to improve both usability and performance.

Key takeaways

• Enablement came first. The Arm CPU path now benefits from pre-built wheels and Docker images, fixes for crashes and accuracy issues, better threading and CPU utilization, and support for chunked prefill and prefix caching. The post also mentions INT8 W8A8 and W4A8 inference support, plus model enablement for GPT-OSS, Whisper, and Qwen 3.5 / 3.6.
• The main problem was not just GEMM. Initial profiling in October 2025 showed that roughly 80% of model runtime was spent in dense layers dispatched to optimized BF16 GEMMs. But standalone GEMM kernels were already close to expected efficiency, so the bigger end-to-end bottlenecks were elsewhere: allocation, synchronization, framework overhead, attention, and quantized execution.
• Allocator behavior had a large impact. PyTorch’s use of glibc malloc on Arm led to poor reuse of large allocations, page faults, and contention under multi-threaded inference. Enabling mimalloc as the default allocator for Arm-based CPUs in PyTorch improved Llama 3.1 8B offline throughput by 2.3x and delivered about 7x gains in low-concurrency serving scenarios.
• High-core-count scaling needed better atomics. Profiling showed that 74% of paged attention time could be spent in OpenMP dynamic scheduling. On Neoverse V2, Arm Large System Extensions provide hardware atomic instructions, but the runtime used by PyTorch did not take advantage of them. A libgomp build using LSE atomics improved Llama 3.1 8B offline throughput by 9% and reduced low-concurrency TPOT latency by 15%.
• Dense layers benefited from weight prepacking. Efficient GEMM kernels prefer blocked weight layouts. Without prepacking, inference repeatedly pays layout conversion costs, especially at low concurrency. The optimized oneDNN path, accelerated by the Compute Library for Arm Architecture, packs BF16 weights during model warmup and reuses them during inference. This raised offline throughput by 16% and reduced low-concurrency TPOT by 60%.
• Paged attention was rewritten for Arm. The previous CPU paged attention path fell back to reference implementations for QK, PV, and softmax exponential work. The new implementation uses custom GEMM kernels with Arm BFMMLA Advanced SIMD instructions and a fast vectorized third-degree polynomial approximation for softmax exponential. Paged attention became up to 4x faster and Llama 3.1 8B offline throughput improved by 12%.

Why it matters

The broader message is that CPU inference performance is an end-to-end systems problem. A fast matrix multiply kernel is necessary, but not sufficient. Allocators, thread runtimes, tensor layout, attention kernels, and quantized paths can dominate real serving behavior.

For enterprises, these changes make Arm CPU servers more credible for cost-sensitive, low-concurrency, or infrastructure-reuse scenarios. For the open-source ecosystem, the work is also notable because many improvements land across shared layers such as PyTorch, oneDNN, OpenMP runtime components, and vLLM itself. That increases the chance that future Arm CPU deployments can benefit without relying on fragile local tuning.

Source: vLLM Blog