Enterprise AI Tokenomics: Why the Bill Outgrows the Demo

Prices and model names in this article were updated on 31 October 2026.

The price of a given level of model quality has fallen fast. According to a16z (November 2024), "What cost $60/million tokens in 2021 costs $.06/million tokens today" for a model of equivalent performance. Epoch AI (March 2025) measured price drops of 9x to 900x per year, depending on the capability milestone. Yet enterprise AI bills keep growing, because cheap tokens do not stop a system from sending millions of them it never needed.

The flagship tier has fallen much less. OpenAI's GPT-4 cost $30 per 1M input tokens and $60 per 1M output in March 2023. GPT-6 Astra costs $10 and $50 as of October 2026. Claude 3 Opus cost $15 / $75 in March 2024, and Claude Opus 5.5 costs $4 / $20. Teams that always pick the newest flagship see only part of the decline.

The arithmetic of a surprise is simple. At Claude Sonnet 5.5's input price of $2 per 1M tokens, 14 billion input tokens a month cost $28,000. Budgets are set from the demo, and a demo never sends 14 billion tokens.

The demo-to-production gap

A demo that summarises a document sends one request: 2,000 tokens in, 500 out. On Claude Sonnet 5.5 ($2 input, $10 output per 1M) that is $0.004 + $0.005 = $0.009. The product manager approves it. Then it ships, and the real usage pattern emerges.

Once live, the same feature sends more on every call. The document now travels with a 1,500-token system prompt, 800 tokens of conversation history and 200 tokens of tool definitions. Input grows from 2,000 to 4,500 tokens, 2.25x the demo. The call now costs $0.009 + $0.005 = $0.014, about 1.6x the demo price. Retries, validation loops and error handling come on top.

Many teams find this out from the first invoice. The LLM cost audit checklist shows how to find it in the logs earlier.

Token maxing

Many developers set max_tokens to the model's ceiling because they are not sure how long the answer should be. The model rarely uses the full allowance. On the occasions it runs long, you pay for every token it writes.

Reasoning models add a second layer. Anthropic's Claude Code docs say thinking tokens "are billed as output tokens". They add that the default thinking budget "can be tens of thousands of tokens per request depending on the model". A generous limit on a reasoning model is a generous limit on the most expensive token type.

The fix is to set max_tokens per endpoint from measured output lengths, a little above the 95th percentile. The model's maximum is the wrong default. Where the model supports it, lower the effort level for simple endpoints. Both changes are free and take effect at once.

Agent loops

One agent task can look like a single user request and still run many LLM calls under the hood. Each call carries the full system prompt again. A 2,000-token system prompt over 30 iterations sends 60,000 tokens of repeats before any work happens.

Here is what a support agent looks like if nobody designs it for cost:

User message: "Why was I charged twice?"

Agent loop iteration 1: system_prompt (2,100 tokens) + user_msg (15 tokens) + tool_definitions (800 tokens) = 2,915 input tokens
Agent loop iteration 2: system_prompt (2,100 tokens) + history (400 tokens) + tool_result (300 tokens) = 2,800 input tokens
Agent loop iteration 3: system_prompt (2,100 tokens) + history (900 tokens) + tool_result (500 tokens) = 3,500 input tokens
...
Total: 30 iterations × ~3,000 avg = 90,000 input tokens for one support ticket

At Claude Sonnet 5.5's input price ($2 per 1M), that is $0.18 per ticket in input alone. At 50,000 tickets a month, it comes to $9,000. The team that budgeted $0.009 per request now has a $9,000 line item for one feature.

Caching only the system prompt changes the picture. Assume one cache write per ticket (2,100 tokens at $2.50 per 1M, $0.00525) and 29 cache reads (60,900 tokens at $0.10 per 1M, $0.00609). The other 27,000 input tokens stay at full price ($0.054). The ticket now costs $0.0653 in input, and 50,000 tickets cost $3,267 a month instead of $9,000.

Running several agents in parallel multiplies all of this. For Claude Code, Anthropic writes that "Agent teams use approximately 7x more tokens than standard sessions when teammates run in plan mode".

Repeated context

Many production AI applications re-send the same content on every request. A legal review tool that re-sends a full contract with every follow-up question pays for that contract again each time. A support bot that includes the product catalogue in every message does the same.

A worked example on Claude Sonnet 5.5: a 15,000-token contract and 20 follow-up questions.

That is 89% less for the same product. Anthropic's cache read on Sonnet 5.5 and Opus 5.5 costs 0.05x the input price. The default cache lives five minutes. A one-hour cache costs 2x the input price to write ($4 per 1M on Sonnet 5.5). The prompt caching guide covers the rules per vendor.

One model for every task

A frequent enterprise mistake is one model for everything. A team tests a strong model and it works. From then on every query goes to it at its price, from "summarise this email" to "write a complex SQL migration".

A "summarise this email" request rarely needs a flagship. GPT-6 Luna costs $0.10 per 1M input tokens; GPT-6.1 Sol costs $2 and GPT-6 Astra $10. If the small model passes your evaluation set for that task, the input price drops 20x against Sol and 100x against Astra.

from openai import OpenAI

client = OpenAI()

# Without routing: everything hits the mid-tier model
response = client.chat.completions.create(
    model="gpt-6.1-sol",
    messages=[{"role": "user", "content": email_text}],
)

# With routing: simple tasks go to the small model
def route_completion(prompt: str, task_type: str):
    model = "gpt-6.1-sol" if task_type == "complex" else "gpt-6-luna"
    return client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
    )

What routing saves, worked out for 1M calls a month at 1,000 input and 300 output tokens each:

The routed setup costs 55% less, including the classifier. The share of traffic a small model can take is specific to your workload; measure it before you count the savings.

The architecture audit checklist

If the AI bill is higher than expected, check these in order:

The pricing drop illusion

Falling token prices did not cut enterprise budgets. They made more ambitious systems affordable, and teams built more agents before fixing the architecture of the first one. Epoch AI also warns that the trend may slow: "The fastest price drops in that range have occurred in the past year, so it's less clear that those will persist." The LLM price history charts how far each tier has fallen since 2023.

Tokens a system never needed to send stay expensive at any unit price.

What to do next

• Log input, cached and output tokens per feature for one week.
• Cache the stable prefix (system prompt, tools, documents) on every high-volume endpoint.
• Route the simplest task type to a small model and measure quality on a fixed evaluation set.
• Before volume grows tenfold, read how costs change at high volume.

Sources

• a16z: Welcome to LLMflation – LLM inference cost is going down fast
• Epoch AI: LLM inference prices have fallen rapidly but unequally across tasks
• OpenAI community forum: GPT-4 API prompt and completion price (2023)
• Simon Willison: Claude 3 launch prices (March 2024)
• Claude pricing (platform.claude.com)
• OpenAI GPT-6 Astra model page
• OpenAI models overview
• Claude Code: manage costs effectively