SiliconBench: Why Local LLM Serving Is More Than a Speed Test

Introduction

Local LLM serving on Apple Silicon is often reduced to a throughput chart: how many tokens per second can an engine produce, and how much does that number improve when more requests arrive? In practice, a desktop serving multiple chat or agent sessions must also preserve memory headroom and maintain output quality. SiliconBench is designed around that broader problem, evaluating nine Apple Silicon engines through speed, memory, and fidelity rather than treating performance as a single number.

Key findings

• A three-dimensional benchmark. The Apple Silicon comparison includes vllm-metal, omlx, llama.cpp, Ollama, mlx_lm, vllm-mlx, SGLang, Hugging Face Transformers, and mistral.rs. vLLM, SGLang, and llama.cpp on DGX Spark provide a complementary serving reference.
• Concurrency depends on architecture. With Qwen3-0.6B, vllm-metal more than doubles throughput for both chat and agent workloads when concurrency increases from 1 to 16. However, CUDA vLLM and SGLang show stronger concurrency scaling on the same prompts, illustrating that an advantage on one device does not automatically transfer to another backend or platform.
• A memory limit is not the same as memory safety. Two stacks complete every evaluated request while approaching physical memory capacity, after which throughput declines. On a unified-memory desktop, model weights, KV cache, and active requests compete for the same resource. A configured budget alone does not guarantee usable headroom.
• Coverage and fidelity are separate requirements. Support for the newer Qwen3.5 and Gemma 4 architectures is narrower. The evaluated implementations match the fidelity reference, yet only three stacks pass all three gates: completing requests, preserving fidelity, and covering the tested models.
• First-token latency exposes scheduling choices. Tests on larger dense and MoE models show that vllm-metal’s packed prefill-decode path maintains lower first-token latency than omlx under concurrent load. The result highlights the need to schedule prompt processing alongside ongoing generation instead of optimizing only aggregate throughput.
• Interconnects shape multi-node scaling. In the tested two-machine configurations, tensor parallelism over Thunderbolt RDMA scales, while pipeline parallelism over TCP regresses.

Why it matters

SiliconBench does not declare one universal winner. Instead, it argues that local serving should be selected by workload. A backend that looks excellent in a single-request test may behave differently when it must share memory among concurrent conversations or agent tasks. Users should therefore verify model support first, measure memory behavior under realistic concurrency, and then check task-level output fidelity rather than relying on tokens per second alone.

For engine developers, the findings point toward more than faster kernels. Better prefill-decode scheduling, transparent memory management, broader support for emerging architectures, and practical multi-node communication are all part of serving readiness. For desktop users, the benchmark offers a more useful decision framework: throughput is important, but it is only one condition for dependable local inference.

Hugging Face Daily Papers