AI Load Balancers: How to Route, Balance, and Govern LLM Traffic at Scale

Once you're making more than a few thousand LLM calls per day across multiple models and providers, you need a layer that handles routing, failover, rate limits, and cost governance. Doing this ad-hoc in application code doesn't scale — you end up with provider-specific logic scattered across services with no central visibility.

Three tools dominate this space: LiteLLM (self-hosted, open source), OpenRouter (managed SaaS aggregator), and Portkey (observability-focused proxy). They're not interchangeable.

LiteLLM: The Open-Source Proxy

LiteLLM provides a unified API over 100+ model providers. Deploy it as a proxy server and all your services call a single endpoint — LiteLLM handles provider SDKs, auth rotation, retry logic, and routing under the hood.

Docker Compose Setup

# docker-compose.yml
version: "3.8"
services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    ports:
      - "4000:4000"
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    command: --config /app/config.yaml --port 4000 --num_workers 8
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - GEMINI_API_KEY=${GEMINI_API_KEY}

  redis:
    image: redis:alpine
    ports:
      - "6379:6379"

# litellm_config.yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: gpt-4o
      api_key: os.environ/OPENAI_API_KEY
      rpm: 500       # rate limit: requests per minute
      tpm: 2000000   # rate limit: tokens per minute

  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
      rpm: 200
      tpm: 800000

  - model_name: gemini-flash
    litellm_params:
      model: gemini/gemini-2.5-flash
      api_key: os.environ/GEMINI_API_KEY
      rpm: 1000
      tpm: 4000000

router_settings:
  routing_strategy: latency-based-routing
  enable_pre_call_checks: true

litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: redis
    port: 6379
    ttl: 600  # 10 minutes

general_settings:
  master_key: ${LITELLM_MASTER_KEY}
  database_url: ${DATABASE_URL}  # PostgreSQL for audit logs

With this config, your services call http://litellm:4000/v1/chat/completions using standard OpenAI SDK syntax. No provider-specific code anywhere else.

Per-Team Budget Controls

LiteLLM's virtual keys let you assign spending limits by team or service:

# Create a virtual key for the "support-bot" team with a $500/month budget
curl -X POST http://litellm:4000/key/generate \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "key_alias": "support-bot-prod",
    "team_id": "support",
    "max_budget": 500,
    "budget_duration": "monthly",
    "models": ["gemini-flash", "claude-haiku"],
    "tpm_limit": 100000
  }'

When the team hits $500, their key stops working until the next billing cycle. No manual intervention, no surprise overruns.

Fallback Chains

# In litellm_config.yaml
router_settings:
  fallbacks:
    - {"claude-sonnet": ["gpt-4o", "gemini-flash"]}
    - {"gpt-4o": ["claude-sonnet", "gemini-flash"]}
  context_window_fallbacks:
    - {"gpt-4o": ["claude-sonnet"]}  # Claude has larger context window
  num_retries: 3
  request_timeout: 30

If Claude hits a rate limit, requests automatically fall back to GPT-4o. If GPT-4o hits context window limits (the input is too long), fall back to Claude Sonnet which has a larger window. This happens transparently — your application doesn't know which model actually ran.

OpenRouter: Managed Aggregation

OpenRouter is a SaaS layer you call instead of individual provider APIs. No infrastructure to manage. It supports 200+ models across all major providers, handles routing, and adds a small margin (~5–10%) on top of provider costs.

from openai import OpenAI

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key="your-openrouter-key"
)

# Use any model from any provider
response = client.chat.completions.create(
    model="anthropic/claude-sonnet-4-5",
    messages=[{"role": "user", "content": "Hello"}],
    extra_headers={
        "HTTP-Referer": "https://yoursite.com",
        "X-Title": "Your App Name"
    }
)

# Or use OpenRouter's auto-routing
response = client.chat.completions.create(
    model="openrouter/auto",  # OpenRouter picks model
    messages=[{"role": "user", "content": "Summarize this document..."}]
)

OpenRouter's key advantage over LiteLLM is zero ops overhead. No container to run, no config to maintain. The trade-off: less control over routing logic, no private key management, and you're dependent on their uptime.

Use OpenRouter when: you're a small team, you don't want to maintain infrastructure, and the 5–10% markup is worth the operational simplicity.

Use LiteLLM when: you need fine-grained routing control, per-team budget enforcement, private key management, or custom audit logging.

Portkey: Observability-First Proxy

Portkey positions itself as "the AI gateway with observability built in." It's a proxy like LiteLLM but with heavier emphasis on traces, cost analytics, and A/B testing of prompts.

from portkey_ai import Portkey

portkey = Portkey(
    api_key="your-portkey-key",
    virtual_key="anthropic-virtual-key"  # stored in Portkey's vault
)

response = portkey.chat.completions.create(
    model="claude-sonnet-4-5",
    messages=[{"role": "user", "content": "Analyze this data..."}],
    metadata={"user_id": "usr_123", "feature": "report-gen"}
)

Portkey's dashboard surfaces cost per user, cost per feature, latency distributions, and error rates — without you building any logging infrastructure. If your primary pain point is "I don't know where my AI spend is going," Portkey is faster to get value from than self-hosted LiteLLM.

Comparison: Which to Use

Key Features to Actually Use

Provider failover is the most critical feature in any of these tools. Don't hardcode a single provider — always configure fallbacks. Provider outages happen, rate limits get hit, and having an automatic fallback to a second provider means your application stays up.

Rate limit handling. LiteLLM respects per-model TPM/RPM limits and queues or routes around them automatically. Without this, you'll hit 429 errors under load and need to implement retry logic in every service.

Cost budgets per team. Don't leave this until you need it. Set per-service monthly budgets on day one. A runaway agent loop that would otherwise burn $5,000 in a weekend gets stopped at $100 if you configured the limit.

Audit logging. Every LLM call should be logged with timestamp, model, input/output token counts, cost, and request metadata. Portkey does this out of the box. LiteLLM writes to PostgreSQL. Both are queryable for cost audits.