Model Routing: How to Cut AI Costs 80% Without Losing Quality

Most production AI applications use one model for everything. Pick GPT-4o, ship it, done. The problem: GPT-4o at $2.50/M input tokens is 33x more expensive than Gemini 2.5 Flash at $0.075/M. If 70% of your queries are "summarize this," "format this JSON," or "classify this text" — you're paying frontier prices for commodity work.

Model routing fixes this. A classifier sits in front of your LLM calls, decides which model to use, and routes accordingly. Simple tasks go to cheap fast models. Complex tasks go to frontier models. Done right, you won't notice the difference except on your invoice.

What Signals to Route On

Routing decisions can be made on several signals:

Query length. Short queries are usually simple. A 15-word question rarely requires GPT-4o. Anything under 50 tokens can default to a fast, cheap model unless other signals suggest otherwise.

Task type classification. Run a lightweight classifier (or prompt a cheap model with a simple prompt) to tag the query: summarization, classification, extraction, generation, reasoning, code. Map these to model tiers.

Explicit complexity markers. Some systems let users or upstream code tag requests: "complexity": "high". Not elegant but effective.

Keyword matching. If the query contains "analyze," "compare," "explain why," or domain-specific jargon, route to the stronger model.

Output requirements. If structured JSON output is required and schema compliance matters, route to a model with reliable structured output. If it's just formatting, use a cheap model.

Routing Tiers

A simple two-tier system covers most use cases:

Add a third tier for reasoning-heavy tasks (o3-mini, Claude Opus 4) if needed. Keep it simple — two tiers handles 80% of cost savings.

Python Implementation with LiteLLM

LiteLLM provides a unified interface across 100+ model providers. You call one function; it handles provider SDKs, auth, and retry logic underneath.

from litellm import completion

def classify_complexity(query: str) -> str:
    """Quick, cheap classifier to route queries."""
    result = completion(
        model="gemini/gemini-2.5-flash",
        messages=[{
            "role": "user",
            "content": f"""Classify this query as 'simple' or 'complex'.
Simple: summarization, classification, formatting, basic Q&A.
Complex: multi-step reasoning, code generation, nuanced analysis.

Query: {query}

Reply with exactly one word: simple or complex"""
        }],
        max_tokens=5
    )
    return result.choices[0].message.content.strip().lower()


def routed_completion(query: str, system_prompt: str = "") -> str:
    complexity = classify_complexity(query)

    model = (
        "gpt-4o" if complexity == "complex"
        else "gemini/gemini-2.5-flash"
    )

    messages = []
    if system_prompt:
        messages.append({"role": "system", "content": system_prompt})
    messages.append({"role": "user", "content": query})

    response = completion(model=model, messages=messages, max_tokens=1024)

    # Log for cost tracking
    print(f"[route] complexity={complexity} model={model} "
          f"tokens={response.usage.total_tokens}")

    return response.choices[0].message.content

The classifier call itself is cheap — a Gemini Flash call for 5 tokens of output costs essentially nothing. The overhead is trivial against the savings on routed traffic.

More Sophisticated Routing with LiteLLM Router

LiteLLM's Router class handles load balancing, fallbacks, and cost-aware routing natively:

from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "economy",
            "litellm_params": {
                "model": "gemini/gemini-2.5-flash",
                "api_key": "your-google-key"
            }
        },
        {
            "model_name": "frontier",
            "litellm_params": {
                "model": "gpt-4o",
                "api_key": "your-openai-key"
            }
        }
    ],
    routing_strategy="cost-based-routing"
)

# Automatically selects cheapest model that meets requirements
response = router.completion(
    model="economy",
    messages=[{"role": "user", "content": "Summarize this article: ..."}]
)

Set routing_strategy="latency-based-routing" if response time matters more than cost. Set fallbacks=[{"economy": ["frontier"]}] to automatically escalate if the cheap model fails or rate-limits.

OpenRouter: Simpler Alternative

If you don't want to manage your own routing logic, OpenRouter aggregates providers and handles fallbacks transparently. You send one request to openrouter.ai/api/v1, specify a model or a routing preference, and OpenRouter handles provider selection.

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemini-2.5-flash",
    "messages": [{"role": "user", "content": "Summarize: ..."}]
  }'

OpenRouter supports auto-routing with "model": "openrouter/auto" — it picks the best model for cost and quality. Not as tunable as a custom LiteLLM setup, but deployable in 10 minutes.

OpenRouter adds a small margin on top of provider costs (~5–10%), but eliminates the operational overhead of managing multiple API keys and provider SDKs.

Measuring the Impact

Track these metrics before and after implementing routing:

# In your logging middleware
cost_per_model = {
    "gpt-4o": 0,
    "gemini-2.5-flash": 0
}
requests_per_model = {
    "gpt-4o": 0,
    "gemini-2.5-flash": 0
}

# After 1 week, calculate:
# - % requests routed to economy tier (target: 60-75%)
# - cost savings vs all-frontier baseline
# - quality regression rate (via human eval or LLM-as-judge on sample)

In practice: teams that implement basic two-tier routing see 60–80% cost reduction within the first week. The quality regression on the economy tier is typically under 5% for tasks that classifier correctly tags as simple — which is why validation matters.

Don't route blindly. For the first two weeks, log every routed decision and periodically sample the economy-tier outputs against what the frontier model would have produced. Tune the classifier threshold until you're comfortable with the trade-off.

What Routing Doesn't Fix

Routing helps with model selection. It doesn't help with inefficient prompts, unnecessary agent loops, or excessive context. Before you build a routing layer, make sure you've already trimmed your system prompts, tuned your max_tokens, and removed unused tool definitions. Routing a bloated request to a cheap model still burns more tokens than routing a lean request to a frontier model.