LLM Tokens Do Not Cost the Same: SIMPLE Models the Hidden Energy Cost of KV Cache

Generative AI is often priced per token, but that pricing unit hides an important technical fact: tokens are not equally expensive to process. During generation, each new token is added to the KV Cache. Every later decoding step must access a larger history, so the energy cost can rise along a sequence even when the tokens themselves appear identical.

A model tied to architecture

The paper “The SIMPLE Law for Energy Cost of LLM Inference” asks whether inference energy can be estimated from a model’s architecture and sequence lengths, rather than measured only after deployment. The authors derive analytical expressions for computation and memory access, covering both multi-head attention (MHA) and grouped-query attention (GQA). These expressions lead to SIMPLE, a four-parameter law for predicting LLM energy use.

SIMPLE separates inference into three sources of cost: prefill for processing the input, KV Cache access during autoregressive decoding, and the computation required to produce the next token. Its inputs include architectural factors such as hidden dimension, layer count, query-head count, and KV-head count, together with the numbers of input and output tokens. This decomposition is important because it gives the model explanatory power instead of treating energy as an opaque fitted curve.

Main findings

• Across traces from 13 dense LLMs in six model families running on NVIDIA H100 GPUs, SIMPLE reached an R2 of 0.969 and a MAPE of 11.97%. The reported error was about 3.1 to 3.5 times lower than the cited state-of-the-art model, without calibration for each individual LLM.
• Prefill energy grows quadratically with hidden dimension. Large models therefore pay a particularly high energy cost when processing long prompts.
• GQA reduces the number of KV heads and consequently lowers cache-related work. In the reported results, more aggressive grouping reduced per-token energy by up to roughly three times compared with standard MHA.
• More reasoning tokens are not automatically more efficient. As the output grows, the KV Cache expands as well. Beyond an analytically derived minimum-energy output length, energy rises super-linearly. Accuracy may continue to improve, but intelligence per joule peaks at an intermediate budget.

Why it matters for serving

The work turns inference energy into a quantity that can be estimated before deployment from architectural choices and workload characteristics. Model providers could use such estimates to compare serving options, while application developers could set output limits and reasoning budgets with energy in mind. For long-context workloads, optimization should address not only arithmetic throughput but also KV Cache capacity, bandwidth, and access patterns.

The result is not a universal replacement for hardware measurements. The evaluation described in the material covers 13 dense models and H100 GPUs; quantization, batching, kernel implementations, and other accelerators may change the observed costs. SIMPLE is best understood as an architecture-level planning and comparison tool, useful for exposing where inference energy comes from.

Source: Hugging Face Daily Papers