Per-token prices dropped 98% between 2022 and 2026. GPT-4-class inference went from $60/million tokens to under $1. So why did the average enterprise AI budget climb from $1.2M in 2024 to roughly $7M in 2026? Because cheap tokens don't stop you from using millions of them badly.
The math is simple: $0.50 per million tokens × 14 billion tokens/month = $7,000. Nobody budgeted for 14 billion tokens. They budgeted for the demo.
The Demo-to-Production Gap
A demo that summarizes a document sends one request: 2,000 tokens in, 500 out. Cost: $0.001. Your product manager approves it. Then it ships and the real usage pattern emerges.
In production, the same feature sends the document plus a 1,500-token system prompt, plus 800 tokens of conversation history, plus 200 tokens of tool definitions — every single time. Suddenly it's 4,500 tokens in per call, 3x the demo cost, and that's before accounting for retries, validation loops, and error handling.
Most enterprises don't discover this until the first invoice arrives.
Token Maxing: The Silent Budget Killer
Many developers set max_tokens to the model's ceiling — 4,096, 8,192, or higher — because they're not sure how long the response should be. The model rarely uses the full allocation, but when it does, you pay for every token.
Worse: some models (especially older GPT-3.5 deployments) were misconfigured to stream until they hit max_tokens even when the content was done. Teams thought they were saving money by using a cheaper model, but were paying for 4,000 tokens of padding per response.
Fix: set max_tokens to the realistic upper bound for each endpoint, not the model's theoretical maximum. A customer service reply rarely needs more than 300 tokens. A code explanation might need 1,500. Setting endpoint-specific limits is free and immediate.
Agent Loops: Where Costs Compound
A single agent task that looks like one user request often involves 20–50 LLM calls under the hood. Each call carries the full system prompt again. If your system prompt is 2,000 tokens and your agent runs 30 iterations, that's 60,000 tokens just in repeated system prompt overhead — before any actual work happens.
Here's what a real support agent architecture looks like if you don't design for cost:
User message: "Why was I charged twice?"
Agent loop iteration 1: system_prompt (2,100 tokens) + user_msg (15 tokens) + tool_definitions (800 tokens) = 2,915 input tokens
Agent loop iteration 2: system_prompt (2,100 tokens) + history (400 tokens) + tool_result (300 tokens) = 2,800 input tokens
Agent loop iteration 3: system_prompt (2,100 tokens) + history (900 tokens) + tool_result (500 tokens) = 3,500 input tokens
...
Total: 30 iterations × ~3,000 avg = 90,000 input tokens for one support ticket
At GPT-4o pricing ($2.50/M input), that's $0.225 per ticket. At 50,000 tickets/month: $11,250. The team that budgeted $0.001 per request is staring at an $11K line item for one feature.
Repeated Context: The Invisible Overhead
Most production AI applications re-send the same content on every request. A legal document review tool that re-sends a 10,000-word contract on every follow-up question is burning tokens you've already paid for. A customer service bot that includes your full product catalog in every message is doing the same.
This isn't a hypothetical. One mid-sized SaaS company was spending $40K/month on a document Q&A feature. Audit revealed 78% of their input tokens were the same document being resent. Implementing server-side caching with Anthropic's cache_control breakpoints cut their bill to $9K the next month — without changing the product at all.
No Routing Logic: Paying Frontier Prices for Commodity Tasks
The most common enterprise mistake is using one model for everything. Teams pick GPT-4o or Claude Sonnet because they tested it and it worked. Then every query — from "summarize this email" to "write a complex SQL migration" — hits the frontier model at frontier prices.
A "summarize this email" request does not need a frontier model. Gemini 2.5 Flash ($0.075/M input) handles it just as well as GPT-4o ($2.50/M input) — that's a 33x cost difference for identical output quality on a simple summarization task.
# Without routing: everything hits GPT-4o
response = openai.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": email_text}]
)
# With routing: simple tasks go to cheap model
from litellm import completion
def route_completion(prompt, task_type):
model = "gpt-4o" if task_type == "complex" else "gemini/gemini-2.5-flash"
return completion(model=model, messages=[{"role": "user", "content": prompt}])
Teams that implement even basic routing — complex vs. simple classification — typically cut costs 60–80% within a week.
The Architecture Audit Checklist
If your AI bill is higher than expected, check these in order:
System prompt size. Paste your system prompt into a tokenizer. Anything over 1,000 tokens on a high-volume endpoint needs a rewrite or caching.
Context window growth. Log the input token count for each request. If it grows linearly with conversation length and you're not truncating, you're paying for the full history every time.
Model assignment. List every LLM call in your app. How many of them actually require a frontier model? Be honest.
Output length. Check your max_tokens settings. Are they matched to realistic output lengths per endpoint?
Retry logic. Are failed requests being retried with the full context? Exponential backoff with truncated context is cheaper than full retries.
The Pricing Drop Illusion
Here's the uncomfortable truth: the 98% price drop in LLM inference didn't save enterprise budgets — it enabled enterprise teams to build more ambitiously without fixing their architecture. Developers who previously couldn't afford to build an agent loop now built five of them, all poorly optimized.
Cheap tokens are a gift. But tokens you never needed to send in the first place are always expensive, regardless of the per-unit price. The enterprises controlling their AI spend in 2026 aren't the ones who found the cheapest model — they're the ones who fixed their architecture before they scaled.