Prices and model names in this article were updated on 31 October 2026.

Pure local inference is heavy to operate and limited in quality. Pure cloud inference gets expensive at scale if every request goes to a frontier model. A hybrid setup sits between them. A local model answers most requests, and a cloud model takes only the ones that need it.

The saving depends on what the cloud alternative costs. The cheap cloud tier costs $0.10 per million input tokens in October 2026 (GPT-6 Luna, Claude Haiku 5.5). Run the numbers in the worked example below before you buy anything. The local versus cloud comparison has the break-even for a single GPU.

The architecture

The setup has three parts:

  1. A local inference server (Ollama) running Llama 3.1 8B or gpt-oss-20b on your own hardware.
  2. Routing logic that labels each request and sends it to local or cloud.
  3. A cloud API (OpenAI, Anthropic or Google) for complex tasks, quality-sensitive paths, and as a fallback when the local server fails.

Ollama serves an OpenAI-compatible endpoint at http://localhost:11434/v1/, and LiteLLM has a built-in Ollama provider. Your application code talks to one router and never needs to know which side answered.

Hybrid setup: a router sends each request to a local model or a cloud model, with cloud as the fallback Request Router rules + task type (LiteLLM Router) easy hard Local (Ollama) llama3.1:8b, gpt-oss:20b no per-token bill Cloud API gpt-6-luna, gpt-6.1-sol $0.10 to $2 per 1M input fallback on error
The router sends easy requests to the local model and hard ones to the cloud; if the local server fails, the cloud model takes over.

Setting up the stack

Step 1: Install Ollama and pull models

# Install Ollama from ollama.com, then pull two local models
ollama pull llama3.1:8b     # general tasks, 4.7 GB at 4-bit
ollama pull gpt-oss:20b     # larger model, fits in 16 GB of memory

# Check that the OpenAI-compatible API is up
curl http://localhost:11434/v1/models

Llama 3.1 8B is a 2024 model. It is small and widely hosted, so you can price it against hosted copies of the same weights. gpt-oss-20b (OpenAI, Apache 2.0) is the larger option if your machine has the memory. Give it the code and structured-output share, then test both on your own examples. The small language models guide lists newer small models worth testing, such as Gemma 4 and Ministral 3.

Step 2: Configure the LiteLLM Router

import os
from litellm import Router

router = Router(
    model_list=[
        # Local Ollama models (LiteLLM recommends the ollama_chat/ prefix)
        {
            "model_name": "local-general",
            "litellm_params": {
                "model": "ollama_chat/llama3.1:8b",
                "api_base": "http://localhost:11434"
            }
        },
        {
            "model_name": "local-code",
            "litellm_params": {
                "model": "ollama_chat/gpt-oss:20b",
                "api_base": "http://localhost:11434"
            }
        },
        # Cloud, frontier tier: $2 / $10 per 1M tokens
        {
            "model_name": "cloud-frontier",
            "litellm_params": {
                "model": "openai/gpt-6.1-sol",
                "api_key": os.environ["OPENAI_API_KEY"]
            }
        },
        # Cloud, cheap tier: $0.10 / $0.50 per 1M tokens
        {
            "model_name": "cloud-mid",
            "litellm_params": {
                "model": "openai/gpt-6-luna",
                "api_key": os.environ["OPENAI_API_KEY"]
            }
        }
    ],
    fallbacks=[
        {"local-general": ["cloud-mid"]},
        {"local-code": ["cloud-frontier"]}
    ],
    allowed_fails=2,
    cooldown_time=60  # seconds before a failed deployment is tried again
)

The fallbacks list does the safety work. If Ollama goes down or runs out of memory, requests move to the cloud model without anyone stepping in. To use Anthropic or Google on the cloud side, swap in their model ids with LiteLLM's provider prefix for that vendor. Claude Haiku 5.5 costs the same as GPT-6 Luna for prompts up to 100K tokens.

The routing logic

A simple rule-based classifier:

import re

COMPLEX_SIGNALS = [
    r'\b(analyze|compare|evaluate|critique|synthesize)\b',
    r'\b(multi-step|step-by-step|chain of thought)\b',
    r'\b(architecture|design|system)\b',
    r'\b(debug|error|exception|stacktrace)\b'
]

SIMPLE_SIGNALS = [
    r'\b(summarize|summary|brief|tldr)\b',
    r'\b(classify|categorize|label|tag)\b',
    r'\b(extract|parse|format|convert)\b',
    r'\b(translate|rewrite)\b'
]

def route_request(prompt: str, task_type: str = None) -> str:
    """Returns the router model name for this request."""

    # Explicit task type override
    if task_type == "complex":
        return "cloud-frontier"
    if task_type == "simple":
        return "local-general"

    # Length heuristic: very short prompts rarely need frontier models
    if len(prompt.split()) < 30:
        return "local-general"

    # Signal matching
    prompt_lower = prompt.lower()
    simple_score = sum(1 for p in SIMPLE_SIGNALS if re.search(p, prompt_lower))
    complex_score = sum(1 for p in COMPLEX_SIGNALS if re.search(p, prompt_lower))

    if complex_score > simple_score:
        return "cloud-frontier"
    elif "code" in prompt_lower or "function" in prompt_lower:
        return "local-code"
    else:
        return "local-general"


def hybrid_completion(prompt: str, task_type: str = None) -> str:
    route = route_request(prompt, task_type)

    response = router.completion(
        model=route,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=1024
    )

    # Log for monitoring: the route asked for and the model that answered
    used_model = response.model
    tokens = response.usage.total_tokens
    print(f"[hybrid] routed_to={route} actual_model={used_model} tokens={tokens}")

    return response.choices[0].message.content

The keyword lists are a starting point. Replace them with labels from your own logs once you have a few hundred examples. The model routing guide covers classifier-based routers.

What to keep local and what to send to the cloud

Keep local: - Email and document summarisation - Text classification and tagging - Data extraction from structured or semi-structured text - Template filling - Simple Q&A against retrieved context - Format conversion (Markdown to HTML and similar) - Short copy such as subject lines and headlines

Send to the cloud: - Complex code generation (new features, architecture decisions) - Multi-document synthesis - Tasks that need recent knowledge (local models stop at their training cutoff) - High-stakes content (legal, medical, financial) where accuracy matters most - Long-form content with complex structure - Anything you ship to end users without review

Grey zone (test with evals): - Code debugging - Customer-facing chat (depends on your quality bar) - SQL generation for complex schemas

Measuring your local hit rate

Add metrics to see how much traffic stays local and what it saves:

from collections import defaultdict

class HybridMetrics:
    def __init__(self, cloud_price_per_m: float = 5.20):
        # Blended cloud price per 1M tokens for the traffic you keep local.
        # 5.20 = GPT-6.1 Sol at 60% input / 40% output: 0.6 x $2 + 0.4 x $10.
        # Use 0.26 if the alternative is GPT-6 Luna: 0.6 x $0.10 + 0.4 x $0.50.
        self.cloud_price_per_m = cloud_price_per_m
        self.calls = defaultdict(int)
        self.tokens = defaultdict(int)
        self.latency = defaultdict(list)

    def record(self, route: str, tokens: int, latency_ms: float):
        tier = "local" if route.startswith("local") else "cloud"
        self.calls[tier] += 1
        self.tokens[tier] += tokens
        self.latency[tier].append(latency_ms)

    def report(self, days: float):
        total_calls = sum(self.calls.values()) or 1
        local_pct = self.calls["local"] / total_calls * 100

        # Cloud spend avoided by answering locally (before local running costs)
        avoided = self.tokens["local"] / 1_000_000 * self.cloud_price_per_m
        per_month = avoided / days * 30

        print(f"Local hit rate: {local_pct:.1f}%")
        print(f"Cloud spend avoided, per 30 days: ${per_month:.2f}")
        for tier in ("local", "cloud"):
            if self.latency[tier]:
                avg = sum(self.latency[tier]) / len(self.latency[tier])
                print(f"Avg {tier} latency: {avg:.0f} ms")

Subtract the local box's running cost from the avoided spend before calling it a saving. There is no universal target for the hit rate. Your eval sets the ceiling: the share of traffic where the local model matched the cloud model on your metric. A hit rate far above that ceiling means hard requests are staying local and quality is slipping.

The cost math, worked through

Take a document-processing workload of 3M tokens a day, 1.8M input and 1.2M output. Suppose an eval shows that 78% of the traffic is easy enough for a small model.

All cloud on GPT-6.1 Sol ($2 / $10):

Hybrid with a local box. The 22% that stays on GPT-6.1 Sol costs 0.22 × $15.60 = $3.43 a day, or $102.96 per 30 days. The box is one RTX 5090, at $1,999 over three years plus 575 W around the clock at 18.34 cents per kWh. That comes to $4.36 a day, or $130.80 per 30 days.

Hybrid with a cheap cloud model instead. Send the same 78% (2.34M tokens a day) to GPT-6 Luna at a blended $0.26 per million. That costs 2.34 × $0.26 = $0.61 a day, or $18.25 per 30 days.

Setup Cloud, 30 days Local box, 30 days Total vs all cloud
All GPT-6.1 Sol $468.00 none $468.00 baseline
78% local (RTX 5090), 22% Sol $102.96 $130.80 $233.76 −50%
78% GPT-6 Luna, 22% Sol $121.21 none $121.21 −74%

Both hybrids cut the bill by half or more. At this volume, though, the cheap cloud tier beats the local box for the easy share. The easy traffic would need to reach about 17M tokens a day before one owned RTX 5090 costs less than GPT-6 Luna. The cheapest LLM APIs article compares the cheap tier by task.

The local side still wins in three cases:

Quality matters as much as price. GPT-6 Luna and a local Llama 3.1 8B are different models, so run your eval against both before you pick. Judge the local model only on the requests you route to it.

What to do next

Pull a week of logs and label each request easy or hard. Run your eval on the easy share with a local model and with GPT-6 Luna or Claude Haiku 5.5. Price both hybrids with the table above, using your own token counts. If the numbers are close, we can help you test the routing at Divero. The October 2026 pricing snapshot has every current price with its source.

Sources