Why Inference, Not Training, Is AI's Real Long-Term Cost

Why does inference cost more than training, despite training being a one-time computation?

GPT-4's training run was a single, bounded computation estimated at 2.1e25 FLOPCITE:E1, whereas Amazon Web Services (AWS) notes that inference recurs continuously after deployment and typically makes up most of the resource usage across the ML lifecycleCITE:E2. Training happens once per model; every subsequent prediction — every chat reply, every API call — triggers a new inference pass, which is why the cost structure tilts toward the deployment phase rather than the initial build.

What share of total AI model costs does inference represent?

AWS estimates that inference — running a trained model to make a prediction — makes up about 90% of the cost of an ML modelCITE:E3. That figure, published by AWS on 2022-04-27, is the clearest quantified anchor for why the operating phase, not the training phase, dominates total spend over a model's life.

Why is LLM inference so expensive? How do memory bandwidth and KV cache drive the cost?

Large language model inference is memory-bound: a paper published 2025-04-08 (arXiv:2504.06319, Dong et al.) finds that decoding is constrained by High Bandwidth Memory (HBM) bandwidth, making LLM inference highly sensitive to memory speed and latencyCITE:E4. A related analysis published 2026-03-20 by Dell Technologies (arXiv:2603.20397) adds that the KV cache's memory footprint scales linearly with context length, imposing critical bottlenecks on GPU memory capacity, memory bandwidth, and inference throughputCITE:E6. Together these two findings point to the same mechanism: as conversations or generated outputs get longer, the KV cache grows, memory pressure rises, and inference — already the larger cost bucket per AWS's estimate — gets more expensive to serve at scale.

How is the industry optimizing inference cost? How much improvement can custom ASICs deliver?

AWS built the Inferentia2 chip specifically for inference economics: announced 2023-04-13, AWS describes its EC2 Inf2 instances as the most cost-effective and energy-efficient option on EC2 for deploying generative AI, delivering up to 50% better performance per watt than other comparable EC2 instancesCITE:E5. That performance-per-watt framing — rather than a raw-throughput claim — lines up with the memory-bound, KV-cache-driven cost structure described above: purpose-built inference silicon targets the bottleneck that actually drives the bill.

How is inference cost trending over the long run? Is per-token pricing falling?

Per-token pricing has fallen steeply: Stanford HAI's AI Index Report 2025 finds that querying a model matching GPT-3.5's MMLU score of 64.8 cost $20.00 per million tokens in November 2022, versus $0.07 per million tokens by October 2024 via Gemini-1.5-Flash-8B — a decline of more than 280-fold in roughly 18 monthsCITE:E7.

What this means

The facts line up into a consistent picture rather than a contradiction: AWS's 90% inference-cost estimateCITE:E3 and its description of inference as recurring resource usage across the ML lifecycleCITE:E2 explain why the memory-bound, KV-cache-driven bottlenecks documented in the 2025 and 2026 papersCITE:E4CITE:E6 matter economically, not just technically. AWS's own response — a chip optimized for performance per watt rather than raw computeCITE:E5 — targets that specific bottleneck. And the 280-fold token-price decline Stanford HAI documents between November 2022 and October 2024CITE:E7 shows the industry has driven down the cost of a fixed-capability query even while GPT-4's training-scale computationCITE:E1 and inference's structural dominance of total costCITE:E3 continue to define where the long-term spending actually sits.