Prices and model names in this article were updated on 31 October 2026.

Past a few thousand LLM calls a day across several models and providers, you need a layer for routing, failover, rate limits and spending control. Ad hoc code in each application does not scale. Provider-specific logic ends up scattered across services, and nobody sees the whole picture.

Three tools cover most of this job, and they are not interchangeable. LiteLLM is an open-source proxy you host. OpenRouter is a hosted aggregator. Portkey is a gateway built around logs and analytics, hosted or self-run.

Services call one gateway, which applies keys, budgets, routing, caching and logging, then forwards to Anthropic, OpenAI or Google, with fallback between providers Support bot Report service Agent jobs Gateway Keys and budgets Rate limits Routing, fallback Response cache Logs with cost Anthropic OpenAI Google Dashed: fallback to another provider when the first returns errors or hits its rate limit.
One endpoint for every service; the gateway owns keys, limits, routing and the cost log.

LiteLLM as a self-hosted proxy

LiteLLM gives one API over many model providers; its routing documentation says it can "call 100+ LLMs". Deployed as a proxy server, it becomes the single endpoint all your services call. It handles the provider SDKs, keys, retries and routing behind that endpoint.

Docker Compose setup

LiteLLM's deployment guide recommends pinning a version tag rather than latest. Keys, budgets and spend logs need PostgreSQL, so the stack includes it.

# docker-compose.yml
services:
  litellm:
    image: docker.litellm.ai/berriai/litellm:v1.90.2   # pin a version
    ports:
      - "4000:4000"
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    command: --config /app/config.yaml --port 4000
    environment:
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - DATABASE_URL=postgresql://litellm:${PG_PASSWORD}@postgres:5432/litellm
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - GEMINI_API_KEY=${GEMINI_API_KEY}

  postgres:
    image: postgres:16
    environment:
      - POSTGRES_USER=litellm
      - POSTGRES_PASSWORD=${PG_PASSWORD}
      - POSTGRES_DB=litellm

  redis:
    image: redis:alpine

# litellm_config.yaml
model_list:
  - model_name: frontier
    litellm_params:
      model: anthropic/claude-opus-5-5
      api_key: os.environ/ANTHROPIC_API_KEY
      rpm: 1000        # Anthropic Start tier: 1,000 requests per minute
      tpm: 2000000     # Start tier: 2,000,000 input tokens per minute

  - model_name: mid
    litellm_params:
      model: openai/gpt-6.1-sol
      api_key: os.environ/OPENAI_API_KEY
      rpm: 5000        # OpenAI Build tier: 5,000 requests per minute
      tpm: 1000000     # Build tier: 1,000,000 tokens per minute

  - model_name: economy
    litellm_params:
      model: gemini/gemini-3.8-flash
      api_key: os.environ/GEMINI_API_KEY
    # Gemini per-model limits are shown in Google AI Studio, not on the docs page

router_settings:
  routing_strategy: latency-based-routing
  enable_pre_call_checks: true   # skip deployments whose context window is too small

litellm_settings:
  cache: true                    # response cache in Redis (REDIS_HOST / REDIS_PORT)
  fallbacks: [{"frontier": ["mid", "economy"]}, {"mid": ["frontier", "economy"]}]
  num_retries: 3
  request_timeout: 30

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL   # keys, teams, users, spend logs

The rpm and tpm values are your own account limits, copied from the vendor's tier table. The ones above are the published defaults for Anthropic's Start tier and OpenAI's Build tier as of October 2026. Replace them with your tier's figures.

Your services then call http://litellm:4000/v1/chat/completions with standard OpenAI SDK syntax. No provider-specific code is needed anywhere else.

Per-team budget controls

LiteLLM's virtual keys carry spending limits per team or service:

# A virtual key for the support bot, $500 budget reset every 30 days
curl -X POST http://litellm:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "key_alias": "support-bot-prod",
    "team_id": "support",
    "max_budget": 500,
    "budget_duration": "30d",
    "models": ["economy", "mid"],
    "tpm_limit": 100000
  }'

LiteLLM's documentation states that once a key crosses its max_budget, "requests fail". The limit resets at the end of budget_duration; durations are written as 30s, 30m, 30h or 30d. Budgets are checked against spend stored in the database, so they need the PostgreSQL service above.

Fallback chains

The fallbacks line in litellm_settings sets the escalation order. If the frontier group errors or hits its rate limit after the retries, requests go to the mid group, then to economy. The application never sees which model answered, so log it. Turning on enable_pre_call_checks also makes the router skip any deployment whose context window is smaller than the request.

Context size is no longer a reason to switch vendors between these models. Claude Opus 5.5 and Sonnet 5.5 take 1M tokens at standard pricing, and GPT-6.1 Sol 1.05M. Fallback between them is mainly about availability and rate limits.

OpenRouter as a hosted aggregator

OpenRouter is a hosted service you call instead of individual provider APIs. There is no infrastructure to run.

from openai import OpenAI

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key="your-openrouter-key"
)

# Any listed model from any provider
response = client.chat.completions.create(
    model="openai/gpt-6.1-terra",
    messages=[{"role": "user", "content": "Hello"}]
)

# Or let OpenRouter's Auto Router pick the model
response = client.chat.completions.create(
    model="openrouter/auto",
    messages=[{"role": "user", "content": "Summarize this document..."}]
)

According to OpenRouter's documentation, the Auto Router classifies each prompt into one of about 30 task types. It then ranks models by what OpenRouter users spend on for that type over the last 7 days. It charges no fee of its own; you pay the chosen model's price.

On cost, OpenRouter's FAQ says it passes through provider prices "without any markup" on inference. A fee applies when you buy credits. You can also bring your own provider keys. Pay-as-you-go accounts get $25,000 a month of such usage with no fee. Above that, they pay 5% of what the same model and provider would cost on OpenRouter.

OpenRouter's advantage over LiteLLM is that there is nothing to operate. You give up some control over routing logic. All traffic also passes through a third party, and you depend on its uptime.

OpenRouter suits a small team that doesn't want to run infrastructure and wants one bill across vendors. LiteLLM suits a team that needs fine-grained routing, per-team budgets, provider keys that stay on its own servers, or its own audit log.

Portkey for logs and cost analytics

Portkey is a gateway with an emphasis on logs, traces and cost analytics. Its plan comparison page lists an open-source gateway you can run locally, a free Dev plan and paid hosted plans.

from portkey_ai import Portkey

client = Portkey(
    api_key="your-portkey-key",   # or the PORTKEY_API_KEY env var
    provider="@anthropic-prod"    # provider slug from Portkey's Model Catalog
)

response = client.with_options(
    metadata={"_user": "usr_123", "feature": "report-gen"}
).chat.completions.create(
    model="claude-opus-5-5",
    messages=[{"role": "user", "content": "Analyze this data..."}]
)

Metadata attached per request lets you break cost down by user or feature in Portkey's dashboard, without building that logging yourself. If your main problem is "I don't know where my AI spend goes", a hosted log is the quickest way to an answer. The LLM cost audit lists the fields to log, whichever tool you choose.

Which one to use

LiteLLM OpenRouter Portkey
Where it runs Your servers OpenRouter's service Hosted, or open-source gateway on your servers
Open source Yes No Gateway yes
Price Free software; you pay for hosting Provider prices without inference markup; fee on credit purchases; BYOK 5% above $25,000 a month Dev free (10,000 requests a month, 3-day log retention); Pro $49 a month (100,000 requests, then $9 per extra 100,000 up to 3M; 30-day retention); Enterprise custom
Per-key budgets Yes, needs PostgreSQL Not covered in the pages we checked Listed with key management on Enterprise
Response caching Redis and semantic cache types Not covered in the pages we checked Simple cache (1-day TTL on Dev, unlimited on paid plans); semantic cache on Pro and Enterprise
Logs and analytics Spend logs in PostgreSQL Not covered in the pages we checked Logs, traces, custom metadata, cost dashboard

Features worth turning on

Provider failover

This one matters most. Don't hard-code a single provider. Outages happen and rate limits get hit. An automatic fallback to a second provider keeps the application up.

Rate limit handling

Give the gateway the rpm and tpm of each deployment. LiteLLM's rate-limit-aware strategy, usage-based-routing-v2, skips deployments that have hit their limit. It sends each request to the deployment with the lowest token use that minute. LiteLLM's routing documentation does not recommend usage-based routing for production because of its performance cost. The weighted default, simple-shuffle, also picks deployments by the rpm or tpm you set. Without these figures, you see HTTP 429 errors under load and need retry logic in every service. Account tiers for each vendor are in LLM costs at high volume.

Cost budgets per team

Set them on day one. A worked example: an agent loop on Claude Opus 5.5 re-sends a 100,000-token context and writes 1,000 tokens per step. Uncached, a step costs 100,000 × $4 / 1M + 1,000 × $20 / 1M = $0.42. At one step every 5 seconds, a weekend (48 hours, 34,560 steps) costs $14,515. A $100 budget on that key stops the loop after about 238 steps, or 20 minutes. Prompt caching cuts the step to $0.04 ($0.02 cached input at $0.20 per million, plus $0.02 output), but the weekend still comes to $1,382.

Audit logging

Log every LLM call with timestamp, model, input and output token counts, cost and request metadata. Portkey records this on its hosted plans; LiteLLM writes spend logs to PostgreSQL. Both can feed a cost audit.

What to do next

Sources