Model Routing: Cut AI Costs Without Losing Quality

Prices and model names in this article were updated on 31 October 2026.

Most production AI applications use one model for everything. Someone picks a flagship, ships it, and nobody revisits the choice. The price gap between a vendor's flagship and its small model is wide as of October 2026. Claude Opus 5.5 costs $4 per million input tokens and $20 per million output. Claude Haiku 5.5 costs $0.10 and $0.50, a 40x difference. OpenAI's GPT-6 Astra ($10 / $50) and GPT-6 Luna ($0.10 / $0.50) are 100x apart. Your traffic may be mostly "summarize this", "format this JSON" or "classify this text". If so, you pay flagship prices for work a small model can do.

Model routing fixes this. A classifier sits in front of your LLM calls, decides which model each request needs, and sends it there. Simple tasks go to a cheap, fast model. Hard tasks go to the flagship. Done well, the difference shows up on the invoice and nowhere else.

What signals to route on

A router can decide on any of the signals below, alone or combined.

Query length

Short queries are usually simple. A 15-word question rarely needs a flagship. Pick a length threshold from your own logs, for example 50 tokens. Below it, requests default to the small model unless another signal says otherwise.

Task type

Run a lightweight classifier, or prompt a cheap model, to tag the query: summarization, classification, extraction, generation, reasoning, code. Map each tag to a model tier.

Complexity tags

Some systems let users or upstream code tag requests with "complexity": "high". Crude, but it works.

Keywords

Send a query to the stronger model if it contains "analyze", "compare", "explain why" or domain jargon.

Output requirements

Some requests need structured JSON where schema compliance matters. Route those to a model you have tested for that. A request that only reformats text can go to the cheap model.

Routing tiers

A two-tier system covers most use cases. Prices are per million tokens, input / output, as of October 2026.

Two caveats on the economy row. Claude Haiku 5.5 moves to $0.50 / $2.50 once a prompt exceeds 100,000 tokens. DeepSeek charges half the peak price off-peak.

A middle tier sits between them. Claude Sonnet 5.5 and GPT-6.1 Sol cost $2 / $10. Gemini 3.8 Flash costs $0.75 / $3.75 until 31 December 2026 and $1.50 / $7.50 from 1 January 2027. Add this tier if your evaluation shows a class of tasks the small model fails and the middle model passes. Most of the price gap lies below the middle tier. Inside Anthropic's line, Opus 5.5 to Sonnet 5.5 is a 2x step and Sonnet 5.5 to Haiku 5.5 is 20x. For more models per tier, see the AI model map and the October 2026 price snapshot.

Python implementation with LiteLLM

LiteLLM is an open-source library that calls many vendors' models through one function (its documentation says "100+ LLMs"). It handles each provider's SDK and authentication underneath.

from litellm import completion

CLASSIFIER = "openai/gpt-6-luna"
ECONOMY = "openai/gpt-6-luna"
FRONTIER = "anthropic/claude-opus-5-5"

def classify_complexity(query: str) -> str:
    """Quick, cheap classifier to route queries."""
    result = completion(
        model=CLASSIFIER,
        messages=[{
            "role": "user",
            "content": f"""Classify this query as 'simple' or 'complex'.
Simple: summarization, classification, formatting, basic Q&A.
Complex: multi-step reasoning, code generation, nuanced analysis.

Query: {query}

Reply with exactly one word: simple or complex"""
        }],
        max_tokens=5
    )
    return result.choices[0].message.content.strip().lower()


def routed_completion(query: str, system_prompt: str = "") -> str:
    complexity = classify_complexity(query)

    model = FRONTIER if complexity == "complex" else ECONOMY

    messages = []
    if system_prompt:
        messages.append({"role": "system", "content": system_prompt})
    messages.append({"role": "user", "content": query})

    response = completion(model=model, messages=messages, max_tokens=1024)

    # Log for cost tracking
    print(f"[route] complexity={complexity} model={model} "
          f"tokens={response.usage.total_tokens}")

    return response.choices[0].message.content

The classifier call costs little. Assume 300 input tokens (instructions plus query) and 5 output tokens on GPT-6 Luna. That is 300 × $0.10 / 1M + 5 × $0.50 / 1M = $0.0000325 per call, or $32.50 per million calls.

Routing with the LiteLLM Router

LiteLLM's Router class groups deployments under a model name and adds load balancing, retries and fallbacks. The routing strategies its documentation lists are simple-shuffle (the default), least-busy, usage-based-routing, latency-based-routing and cost-based-routing. A strategy chooses among the deployments inside one model group. It does not decide which group a request belongs to; your classifier still does that.

from litellm import Router

router = Router(
    model_list=[
        # Two deployments in the "economy" group; cost-based routing picks between them
        {
            "model_name": "economy",
            "litellm_params": {
                "model": "openai/gpt-6-luna",
                "api_key": "your-openai-key"
            }
        },
        {
            "model_name": "economy",
            "litellm_params": {
                "model": "gemini/gemini-3.8-flash",
                "api_key": "your-google-key"
            }
        },
        {
            "model_name": "frontier",
            "litellm_params": {
                "model": "anthropic/claude-opus-5-5",
                "api_key": "your-anthropic-key"
            }
        }
    ],
    routing_strategy="cost-based-routing",
    fallbacks=[{"economy": ["frontier"]}],  # escalate if the cheap group fails
    num_retries=2
)

response = router.completion(
    model="economy",
    messages=[{"role": "user", "content": "Summarize this article: ..."}]
)

Set routing_strategy="latency-based-routing" if response time matters more than price. The fallbacks entry sends a request to the frontier group when every economy deployment fails or hits its rate limit. Load balancing across several gateways is covered in AI load balancers.

OpenRouter as a simpler alternative

If you don't want to run your own routing code, OpenRouter sells access to many vendors' models through one OpenAI-compatible endpoint. You send one request to openrouter.ai/api/v1 and name a model.

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-6-luna",
    "messages": [{"role": "user", "content": "Summarize: ..."}]
  }'

OpenRouter also has an Auto Router, "model": "openrouter/auto". Its documentation says a lightweight classifier assigns each prompt one of about 30 task types. It then ranks models by what OpenRouter users spend on for that task type over the trailing 7 days. A cost_tier setting (low to max) narrows the candidates, and allowed_models restricts them, for example to anthropic/*. The router adds no fee; you pay the selected model's price. Its ranking reflects popularity. It runs no quality test on your data, so treat its pick as a starting point.

On cost, OpenRouter's FAQ says it passes through provider prices "without any markup" on inference. A fee applies when you buy credits. You can also bring your own provider keys. Pay-as-you-go accounts get $25,000 a month of such usage with no fee. Above that, they pay 5% of what the same model and provider would cost on OpenRouter.

Measuring the impact

Track these numbers before and after you add routing:

# In your logging middleware
cost_per_model = {
    "anthropic/claude-opus-5-5": 0.0,
    "openai/gpt-6-luna": 0.0,
}
requests_per_model = {
    "anthropic/claude-opus-5-5": 0,
    "openai/gpt-6-luna": 0,
}

# After 1 week, calculate:
# - share of requests routed to the economy tier
# - cost savings vs an all-frontier baseline
# - quality regression rate (human review or LLM-as-judge on a sample)

The saving depends on one number that only your own traffic can give. That number is the share of requests the small model handles acceptably. Here is a worked example with an assumed 70% share. The workload is 1M queries a month, each with 1,500 input tokens and 400 output tokens.

That is a 68% cut. A shortcut gives the same figure: savings ≈ routed share × (1 − small price ÷ flagship price) = 0.7 × (1 − 1/40) ≈ 68%. At a 40% routed share the same formula gives 39%. The cost audit checklist shows how to find the routing candidates in your logs.

Don't route blindly. For the first two weeks, log every routing decision. Regularly sample economy-tier outputs and compare them with what the frontier model would have produced. Tune the classifier until the error rate is one you accept.

What routing doesn't fix

Routing helps with model selection. It does nothing about inefficient prompts, unnecessary agent loops or excess context. Before you build a routing layer, trim your system prompts, set max_tokens per endpoint and remove unused tool definitions. A bloated request sent to a cheap model can still cost more than a lean one sent to a flagship.

What to do next

• Tag one week of traffic by task type and count how much of it is classification, extraction or formatting.
• Build an evaluation set of 100–200 real requests per task type and run it on one small and one flagship model.
• Start with two tiers; add a middle tier only where the evaluation shows a gap.
• We can review a routing split on a live pipeline; the LLM cost audit checklist shows what we look at.

Sources

• Claude pricing (platform.claude.com)
• OpenAI models overview
• OpenAI GPT-6 Astra model page
• Gemini API pricing
• DeepSeek API pricing
• LiteLLM: Router, load balancing and fallbacks
• OpenRouter FAQ
• OpenRouter Auto Router