Pure local inference is operationally heavy and quality-constrained. Pure cloud inference is expensive at scale. The hybrid approach threads the needle: run a local model for the majority of requests, escalate to a cloud model only when the task genuinely needs it.

Done well, you use cloud APIs for 10–20% of requests and spend 10–20% of what a cloud-only setup would cost.

The Architecture

The setup has three components:

  1. Local inference server (Ollama) running Llama 3.1 8B or Qwen 2.5 7B on local hardware
  2. Routing logic that classifies each request and assigns it to local or cloud
  3. Cloud API fallback (OpenAI, Anthropic, or Gemini) for complex tasks or quality-sensitive paths

Ollama exposes an OpenAI-compatible endpoint at localhost:11434/v1, which means LiteLLM can talk to it identically to any other provider.

Setting Up the Stack

Step 1: Install Ollama and pull models

# macOS/Linux
curl -fsSL https://ollama.com/install.sh | sh

# Pull recommended models for hybrid setup
ollama pull llama3.1:8b        # General tasks, 4.7GB
ollama pull qwen2.5:7b         # Strong on coding/structured tasks, 4.4GB

# Verify the API is up
curl http://localhost:11434/v1/models

Qwen 2.5 7B is worth knowing about — it outperforms Llama 3.1 8B on coding and structured output tasks despite similar size. For hybrid setups doing code review or extraction, Qwen is the better local choice.

Step 2: Configure LiteLLM Router

from litellm import Router

router = Router(
    model_list=[
        # Local Ollama models
        {
            "model_name": "local-general",
            "litellm_params": {
                "model": "ollama/llama3.1:8b",
                "api_base": "http://localhost:11434"
            }
        },
        {
            "model_name": "local-code",
            "litellm_params": {
                "model": "ollama/qwen2.5:7b",
                "api_base": "http://localhost:11434"
            }
        },
        # Cloud fallback
        {
            "model_name": "cloud-frontier",
            "litellm_params": {
                "model": "gpt-4o",
                "api_key": os.environ["OPENAI_API_KEY"]
            }
        },
        # Cloud mid-tier
        {
            "model_name": "cloud-mid",
            "litellm_params": {
                "model": "claude-haiku-3-5",
                "api_key": os.environ["ANTHROPIC_API_KEY"]
            }
        }
    ],
    fallbacks=[
        {"local-general": ["cloud-mid"]},
        {"local-code": ["cloud-frontier"]}
    ],
    allowed_fails=2,
    cooldown_time=60  # seconds to wait before retrying local after failures
)

The fallbacks config is critical. If Ollama goes down or runs out of memory, requests automatically escalate to the cloud model. No manual intervention needed.

The Routing Logic

A simple classifier that routes 80%+ of requests locally:

import re

COMPLEX_SIGNALS = [
    r'\b(analyze|compare|evaluate|critique|synthesize)\b',
    r'\b(multi-step|step-by-step|chain of thought)\b',
    r'\b(architecture|design|system)\b',
    r'\b(debug|error|exception|stacktrace)\b'
]

SIMPLE_SIGNALS = [
    r'\b(summarize|summary|brief|tldr)\b',
    r'\b(classify|categorize|label|tag)\b',
    r'\b(extract|parse|format|convert)\b',
    r'\b(translate|rewrite)\b'
]

def route_request(prompt: str, task_type: str = None) -> str:
    """Returns model name for routing."""

    # Explicit task type override
    if task_type == "complex":
        return "cloud-frontier"
    if task_type == "simple":
        return "local-general"

    # Length heuristic: very short prompts rarely need frontier models
    if len(prompt.split()) < 30:
        return "local-general"

    # Signal matching
    prompt_lower = prompt.lower()
    simple_score = sum(1 for p in SIMPLE_SIGNALS if re.search(p, prompt_lower))
    complex_score = sum(1 for p in COMPLEX_SIGNALS if re.search(p, prompt_lower))

    if complex_score > simple_score:
        return "cloud-frontier"
    elif "code" in prompt_lower or "function" in prompt_lower:
        return "local-code"
    else:
        return "local-general"


def hybrid_completion(prompt: str, task_type: str = None) -> str:
    model = route_request(prompt, task_type)

    response = router.completion(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=1024
    )

    # Log for monitoring
    used_model = response.model
    tokens = response.usage.total_tokens
    print(f"[hybrid] routed_to={model} actual_model={used_model} tokens={tokens}")

    return response.choices[0].message.content

What to Keep Local vs. Send to Cloud

Keep local: - Email/document summarization - Text classification and tagging - Data extraction from structured or semi-structured text - Template filling - Simple Q&A against retrieved context - Format conversion (markdown → HTML, etc.) - Short creative copy (subject lines, headlines)

Send to cloud: - Complex code generation (new features, architecture decisions) - Multi-document synthesis - Tasks requiring recent knowledge (local models have a training cutoff) - High-stakes content (legal, medical, financial) where accuracy is critical - Long-form content with complex structure - Anything you're shipping to end-users without review

Gray zone (test with evals): - Code debugging (7B models are surprisingly good at this for common errors) - Customer-facing chat (depends on your quality bar) - SQL generation for complex schemas

Measuring Your Local Hit Rate

Add metrics to understand how effectively you're using local inference:

from collections import defaultdict
import time

class HybridMetrics:
    def __init__(self):
        self.calls = defaultdict(int)
        self.tokens = defaultdict(int)
        self.latency = defaultdict(list)

    def record(self, model: str, tokens: int, latency_ms: float):
        tier = "local" if "ollama" in model or "llama" in model or "qwen" in model else "cloud"
        self.calls[tier] += 1
        self.tokens[tier] += tokens
        self.latency[tier].append(latency_ms)

    def report(self):
        total_calls = sum(self.calls.values())
        local_pct = self.calls["local"] / total_calls * 100

        # Estimate cost savings
        cloud_token_cost = self.tokens["cloud"] * 0.000004  # ~$4/M blended
        total_token_cost_if_cloud_only = sum(self.tokens.values()) * 0.000004
        savings = total_token_cost_if_cloud_only - cloud_token_cost

        print(f"Local hit rate: {local_pct:.1f}%")
        print(f"Estimated monthly savings: ${savings * 30:.0f}")
        print(f"Avg local latency: {sum(self.latency['local'])/len(self.latency['local']):.0f}ms")
        print(f"Avg cloud latency: {sum(self.latency['cloud'])/len(self.latency['cloud']):.0f}ms")

Target a local hit rate of 70–80%. Below 60% means your classifier is too aggressive about escalating. Above 90% might mean you're keeping complex tasks local and degrading quality.

Real-World Numbers

A document processing platform running this hybrid setup:

The local hardware (2× RTX 3090) was a $1,500 investment that paid off in month 2.

The quality regression on locally-routed tasks was under 3% as measured by human evaluation — because the classification logic was carefully tuned to keep complex tasks in the cloud tier. The key insight: you don't need local models to be as good as GPT-4o overall. You need them to be good enough on the specific tasks you route to them.