Inference used to be an engineering afterthought next to training. That world is gone. The spending center of gravity is shifting to “always-on” inference, where every extra output token is literally more GPU time at high power. When models reason longer and speak more, “better answers” quietly becomes a power procurement problem.
The market shorthand is still “$/M tokens.” The real constraint is “J/token” (and then PUE turns that into facility kWh). Once you map tokens to electricity, two things become obvious: (1) long outputs are not free, and (2) efficiency gains do not just improve margins, they unlock capacity under grid and contract constraints.
[wnr_snapshot title=”Tokens -> electricity (at a glance)” items=”Global data center electricity|~415 TWh (2024)|IEA estimate, ~1.5% of global electricity;2030 data center electricity|>~1,000 TWh (base case)|IEA projection (generation to supply data centers);U.S. data center electricity|~176 TWh (2023)|DOE/LBNL estimate, ~4.4% of U.S. electricity;Industry-average PUE|~1.56 (2024)|Uptime Institute survey;Token scale disclosed|~7B tokens/min|Alphabet cited direct API processing (Q3 2025);Frontier-model energy per token|~163–236 J/token|TokenPowerBench range for a 405B-class model on 16×H100 (varies by workload/parallelism)”]
The translation: from “tokens” to “MW”
If you know (a) energy per token at the server level, and (b) PUE, you can turn any token volume into a power and cost estimate. This is the missing mental model: tokens are not abstract. They are a metered workload.
[wnr_math title=”Quick math: how tokens become a power line item” items=”Tokens/day from 7B tokens/min|10.08T tokens/day|7B × 60 × 24;Energy per 1M tokens|0.278 kWh × (J/token) × PUE|1 kWh = 3.6 MJ;Small-model example|~0.087 kWh per 1M tokens|0.2 J/token, PUE 1.56;Frontier-model example|~86.7 kWh per 1M tokens|200 J/token, PUE 1.56;At 1 J/token (avg) @ 7B/min|~182 MW facility draw|IT ~117 MW × PUE 1.56;At 5 J/token (avg) @ 7B/min|~0.91 GW facility draw|This is why longer outputs become capacity planning”]
Why output tokens are the expensive ones
Output tokens typically live in the “decode” phase: sequential generation where latency and throughput constraints keep the hardware hot. This is where user-facing behavior matters. A model that “thinks longer” and then emits longer outputs can multiply energy per request even if the visible user action (one query) feels unchanged.
TokenPowerBench is a useful anchor because it measures inference power directly across workloads, architectures, and deployment strategies. Three findings matter for operators:
1) Energy per token rises with prompt/context length (more memory traffic, longer attention/KV work).
2) Parallelism strategy changes energy per token materially for frontier-scale models.
3) Utilization is everything: better batching can reduce energy per token meaningfully until you saturate the device.
[wnr_table title=”What actually moves joules per token (operator view)” headers=”Lever|What the evidence shows|Why it matters for inference economics” rows=”Model size|TokenPowerBench: energy/token rises as models scale; Llama-3 1B to 70B was ~7.3x higher energy/token|Bigger models can be qualitatively better, but the per-token energy penalty is real;Prompt and output length|TokenPowerBench: Llama-3 70B energy/token rose ~3x when context went ~2K -> ~10K|Long contexts and long outputs can quietly multiply power per request;Batching and utilization|TokenPowerBench: as batch size rises, GPUs move from 256)|If you can batch, you cut J/token; if you cannot (low latency), you pay;Architecture choice (MoE)|TokenPowerBench: MoE can reduce energy/token ~2–3x vs dense at similar quality (case studies)|Routing fewer active parameters per token is an energy lever, not just a model design detail;Inference engine|TokenPowerBench: swapping frameworks/engines cut energy/token ~25–40% (case studies)|Kernel efficiency becomes a power strategy, not just a speed tweak” notes=”Energy per token varies by model, hardware, parallelism, batch size, and workload. Use these as directional levers, not universal constants.” ]
Electricity cost: usually small… until it isn’t
For small and mid models, the pure energy charge per million tokens can look like rounding error. But two things change the picture fast: frontier-scale inference (high J/token) and power contracting (demand charges, indexed pricing, constraints).
The point is not “electricity dominates every token.” The point is that for the workloads and models that matter most (frontier reasoning, long outputs, low-latency), electricity and capacity become binding constraints.
[wnr_table title=”Electricity-only token bill (illustrative, facility-level)” headers=”Scenario|Energy per token (J, IT)|Facility kWh per 1M tokens (PUE 1.56)|Electricity cost per 1M tokens (8.65¢/kWh)|So what?” rows=”Small dense model (representative)|0.2|0.087|~$0.01|Energy is negligible; capacity and capex dominate;Mid model (representative)|1|0.433|~$0.04|Still small per 1M, but visible at trillion-token scale;405B-class frontier (TokenPowerBench range)|163–236|~70.7–102.3|~$6.11–$8.85|Electricity becomes first-order, and peaks/delivery constraints matter even more” notes=”This is electricity only. It excludes capex, networking, staffing, and the economics of reserved capacity. Electricity price varies widely by region and contract.” ]
The power bill has two parts: energy and peaks
Energy charges scale with kWh. Demand charges scale with your maximum kW draw during the billing period. That distinction matters for inference-heavy operations because user demand can be spiky, and “peak tokens per second” can set the month’s bill even if average usage is lower.
This is where contract shape becomes strategy:
• Fixed-price retail supply dampens volatility but does not remove peak costs.
• Indexed or pass-through pricing can turn a heatwave or congestion event into an inference margin shock.
• Interruptibility and curtailment options are easier for training than for real-time inference (SLA and latency make it harder to shed load).
[wnr_table title=”Power contract exposure: why token growth changes procurement” headers=”Contract element|What it is|How inference behavior hits it|Mitigation idea” rows=”Energy charge (kWh)|Price per unit of electricity, fixed or indexed|More tokens -> more kWh, but J/token rises with longer prompts and bigger models|Model routing, shorter outputs where acceptable, engine efficiency;Demand charge (kW)|Charge based on peak demand in the billing period|Spiky inference can set peak kW and lock in a monthly bill|Load shaping, batching, caching, multi-region routing, storage;Capacity reservation|Paying for deliverable MW and interconnect|Sustained growth forces more MW, long lead times and capex|Earlier procurement, distributed campuses, phased ramps;Curtailment terms|Ability/obligation to reduce load during grid stress|Inference is harder to curtail than training|Separate training from inference pools, precompute where possible” notes=”The biggest surprises usually come from peaks and delivery constraints, not average kWh.” ]
Historical context: data centers left the flatline
For years, the story was “efficiency offsets growth.” In the U.S., total data center electricity consumption was roughly flat around ~60 TWh from 2014–2016, then rose to ~76 TWh in 2018 and ~176 TWh by 2023. Now the forward curve is steep: DOE/LBNL project ~325–580 TWh by 2028. Globally, the IEA estimates ~415 TWh in 2024 and a base-case path to over ~1,000 TWh by 2030.
Inference is why this matters. Training comes in waves. Inference is the daily metronome that can turn token growth into a structural load on grids, and into a structural cost line for operators.
Bottom line
Per-token pricing is how AI is sold. Per-joule reality is how AI is paid for. As outputs get longer and test-time compute becomes a product feature, inference turns into an electricity story: capacity planning, procurement strategy, and efficiency engineering all collapse into the same KPI. The winners will be the operators who can keep joules-per-token falling while tokens-per-user rises.
Sources (primary)
• International Energy Agency (Apr 2025): Energy and AI (global data center electricity 415 TWh 2024; >1,000 TWh 2030 base case; demand growth framing) — https://iea.blob.core.windows.net/assets/601eaec9-ba91-4623-819b-4ded331ec9e8/EnergyandAI.pdf
• Lawrence Berkeley National Laboratory / DOE-supported (Dec 2024): United States Data Center Energy Usage Report (176 TWh 2023; 325–580 TWh 2028; historical series) — https://climate.mit.edu/sites/default/files/2025-02/lbnl-2024-united-states-data-center-energy-usage-report.pdf
• Uptime Institute (Jul 2024): Global Data Center Survey 2024 (industry-average PUE ~1.56, trend commentary) — https://datacenter.uptimeinstitute.com/rs/711-RIA-145/images/2024.GlobalDataCenterSurvey.Report.pdf
• Zhou et al. (arXiv, Dec 2025): TokenPowerBench: Fine-grained power profiling for LLM inference (J/token ranges; batching, context-length, MoE and engine effects) — https://arxiv.org/pdf/2512.03024
• OpenAI (Sep 2024): Learning to reason with LLMs (test-time compute: performance improves with more “time spent thinking”) — https://openai.com/index/learning-to-reason-with-llms/
• Alphabet earnings call transcript (Q3 2025): disclosed ~7B tokens per minute via direct API processing — https://www.alphabet.com/en-us/earnings-call-transcript/
• U.S. EIA (Table 5.3, accessed via EPM): U.S. average electricity price by sector (industrial ~8.65¢/kWh; commercial ~13.41¢/kWh, Oct 2025) — https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_03
• U.S. DOE FEMP: Utility bills and demand charges (how peak kW becomes a bill line) — https://www.energy.gov/femp/articles/understanding-your-utility-bill
Leave a Reply