Two AI requests can generate the same number of answer tokens and place very different demands on a GPU. An ordinary question may include a short prompt. A coding agent can arrive with tens of thousands of tokens of repository context, earlier tool results and conversation history. Every subsequent output token may need attention to that much larger context. A November 2 preprint from researchers at the University of Bologna and CINECA derives a compact way to estimate the associated energy cost—and its worked example has an important surprise for teams relying heavily on prompt caching.[1]
The proposed SIMPLE model separates initial processing, called prefill, from the work of decoding new tokens. It also accounts for the key-value memory cache used by attention mechanisms. Across measured traces for 13 dense language models from six families on a single NVIDIA H100 GPU using TensorRT-LLM, the authors report an R-squared of 0.969 and mean absolute percentage error of about 11.97% for energy prediction. Those are fit statistics on the studied hardware and workloads, not an assurance that every data-centre bill or deployment can be predicted to 12% accuracy.[1]
At a glance
| Measure | Value | Context |
|---|---|---|
| Measured models | 13 | Dense LLMs on H100 hardware |
| Prediction error | ~11.97% | MAPE within evaluated traces |
| No-cache agent estimate | About 13× | Against specified short-context comparator |
| With prefix cache | 4.8× | Estimated remaining energy burden |
Where the 13-times figure comes from
The authors use reported median context characteristics from a coding-agent workload to construct three scenarios for Llama 3.1 8B. A non-agentic comparator has about 5,000 input tokens and 247 output tokens in the specific calculation. The longer agent request carries around 68,000 input tokens and the same output length. Without caching, SIMPLE estimates the short request at 138.1 joules and the long request at 1,787.7 joules—approximately 13 times as much. These are modelled per-request estimates. Because the calibration traces used input lengths of 64–4,096 tokens, the 68,000-token example extrapolates beyond that measured range.[1]
The long request is costly in two ways. More fresh input requires more prefill computation. During output generation, the model must also read and attend to a much larger retained context. Token prices quoted to customers generally do not reveal either component directly. A service that looks cheap on a per-output-token basis may still have a substantial physical footprint when its requests carry extensive histories.[1]
Caching changes the first component. In the paper's agent example, about 63,000 of the 68,000 input tokens can be served from a reused prefix. That reduces the prefill contribution by around 93%. Total estimated energy falls from 1,787.7 to 668.7 joules, a 62.6% reduction calculated from the reported values. The cached request is therefore dramatically cheaper than a fully recomputed long-context request, but it is still 4.8 times the short-context comparator. The reason is that decoding must still attend to the full context, with key-value cache traffic representing most of the remaining cost in this worked example.[1]
Not all tokens should be treated as equal work
The four-parameter SIMPLE equation uses features of model architecture and request geometry rather than fitting a new independent coefficient set for every individual model. Its fitted coefficients remain specific to the tested hardware and serving setup. It covers multi-head and grouped-query attention, and the authors report benefits from grouping that reduces key-value memory demands. The trade-off is understandable: a more general law loses some accuracy against a finely calibrated model-specific fit, but can reveal how design choices change energy before every candidate model is separately measured.[1]
The 13-fold comparison does not account for the entirety of a production service. GPUs may be shared across requests; batching, idle power, speculative decoding, server utilisation, other hardware and cooling can affect the energy allocated to a customer interaction. The study does not establish that all agent requests in the industry consume thirteen times more electricity than all chat requests, nor that caching eliminates the cost of long-lived attention. The authors' arithmetic is conditional on model, output length and the chosen workload geometry.[1]
For an operator building cost and capacity dashboards, the more useful message is to track input context, uncached input, output length, reuse rate, model architecture and actual utilisation together. Cutting unnecessary context may reduce both prefill and decoding costs. Caching large shared prefixes can help, but a product that continually appends logs and tool traces may retain a cost that an invoice measured in undifferentiated tokens conceals. Those measurements would make the components of inference energy visible enough to optimise.[1]
Sources & further reading
- The SIMPLE Law for Energy Cost of LLM Inference Cavagna and colleagues, University of Bologna / CINECA, arXiv · 2 Nov 2026 · Accessed 4 Nov 2026