Why the Same Model Can Feel Dumber Locally: The Inference Stack Matters

Introduction

Why can an identical model checkpoint appear less capable when it runs locally? An experiment by Level1Techs forum user thr3e points to the inference stack rather than the weights themselves. Using Qwen3.6-27B on an RTX PRO 6000 Blackwell GPU, the experiment captured logits across more than 100,000 tokens from a real Agent workflow and compared attention backends, KV-cache precision, weight-quantization methods, and tensor-parallel settings.

A logit is the raw score assigned to each candidate token. In theory, identical weights and identical input should produce identical scores. In practice, floating-point precision, accumulation order, CUDA kernels, and cross-device reductions can create small numerical differences. These differences may be invisible on short prompts, but over a long context they can accumulate and change later decisions.

Key findings

• Attention backends can change generation. With other variables held constant, FlashAttention 2, Flash Inference, and Triton Attention initially produced the same choices. Divergence appeared as the context grew. In one tool-use trace, an interface name was generated incorrectly, leading to further incorrect commands.
• Compressed KV caches are risky for long contexts. BF16 remained stable in the reported test. INT8 showed flips but recovered in the observed case, while INT4 diverged sharply and failed to recover the tool-calling trajectory. Memory savings can therefore come with behavioral instability.
• Lower-bit quantization is not automatically better. A community INT8 W8A16 build showed higher Top-1 agreement than the tested official FP8 and NVFP4 variants. The analysis attributed this partly to retaining BF16 activations and excluding some sensitive layers. This is a test result, not a universal ranking of quantization methods.
• Tensor parallelism can matter. The same BF16 weights behaved differently under TP1, TP2, and TP4. The experiment connected the discrepancy to numerical differences in NCCL cross-GPU reductions.

Why it matters

The checkpoint is only one part of an inference result. A container may include hundreds of Python, CUDA, communication, and compilation components, all of which help determine the execution path. A KL-divergence figure is difficult to interpret unless the reference checkpoint, runtime, context length, sampling positions, calibration data, and aggregation method are disclosed.

This does not make quantization or local deployment useless. It does mean that short question-and-answer tests are insufficient for code execution, tool use, and long-context Agents. Developers should pin versions and configurations, record the attention backend, KV precision, quantization implementation, parallelism, driver, and hardware, then run logit or behavioral regression tests on their own workloads. Model evaluation may increasingly need to describe not only which model is used, but also which inference stack runs it.

Source: QbitAI