Sable Makes Large Language Model Inference Verifiable Bit by Bit

Introduction

A hosted language model normally requires users to trust both the model being run and the computation that produced the answer. Sable, short for Small-Field Argument for Bit-Exact LLM Execution, turns that trust assumption into a verifiable claim. Given committed weights, a prompt, and a response, a verifier can check that the committed model produced every token in the sequence, including the stopping decision, without rerunning inference on the original GPU.

How the system works

Sable does not simply attach a proof layer to ordinary BF16 inference. Instead, it defines a fixed integer function, F_O, with no floating-point step. Rounding, normalization, softmax, and sampling are all determined by their inputs. This gives the prover a discrete and reproducible execution target, avoiding ambiguity caused by floating-point differences across hardware and software stacks.

The constraint system operates over the 31-bit KoalaBear prime field. Its relations are designed so that intermediate values provably do not wrap around the modulus. Sable batches zerochecks and LogUp-GKR lookup checks into one sumcheck per domain. It also commits to model weights and witness tensors through a single multilinear commitment combining Reed-Solomon codes, BLAKE3 Merkle trees, and a mixed-length BaseFold opening.

The prover is designed to stream the model layer by layer on one GPU instead of keeping every intermediate tensor in memory at once. On an NVIDIA RTX PRO 6000 Blackwell, the reported time for Qwen2.5-32B is 92.5 seconds for 4,096 tokens and 398 seconds for 16,384 tokens. For 4,096 tokens, Qwen3-32B takes 121.8 seconds and the mixture-of-experts Qwen3-30B-A3B takes 84.7 seconds. A compiled verifier checks a proof on one CPU core in roughly 19 to 56 seconds.

What the numbers mean

The main practical limitation is still cost. Sable's proof generation takes about 41 to 50 times as long as serving, and the paper estimates a cost of roughly 6.4 cents per 4,096-token request at public GPU prices. Compared with zkLLM, the authors report 40 to 64 times less time per parameter and token. That is a substantial improvement, but it does not make proof-producing inference as cheap as ordinary serving.

Accuracy also needs to be interpreted carefully. F_O is the function of the committed integer-quantized model; it is not a bit-for-bit reproduction of arbitrary BF16 execution. On WikiText-2, its greedy token agrees with the BF16 model at 86.3% of positions. On Qwen3, its results are within 1.6 percentage points on MMLU and 1.7 points on GSM8K. Sable therefore addresses two separate challenges: proving the computation efficiently and ensuring that the provable integer computation preserves the intended model behavior.

Significance and limitations

The work is a notable step toward auditable inference for large models. It could help users verify model versions, inspect outsourced inference, and reduce the need to trust a service operator. However, the guarantee applies to a committed integer model and its precisely defined execution function, not automatically to an original floating-point model or every deployment environment. Longer contexts also increase the proof burden. Faster proving hardware, more efficient quantization, and lower-overhead protocols will determine whether this approach can move from specialized auditing to routine online inference.

Source: Hugging Face Daily Papers