7 Smart Techniques to Slash LLM Costs in Production (Beyond Just Switching Models)

Switching to a cheaper model is the obvious move. After you've done that, these seven techniques are what separate teams running efficient production AI from teams who spend 5x what they need to.

1. Prompt Compression with LLMLingua

LLMLingua (Microsoft Research) removes low-information tokens from your prompts using a small, fast language model. In practice, it can compress context by 3–5x with under 5% quality degradation on downstream tasks.

The technique works by scoring each token for its perplexity contribution and dropping the least informative ones. A 3,000-token document becomes 700–1,000 tokens that preserve the critical information.

from llmlingua import PromptCompressor

compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-bert-large-multilingual-cased-meetingbank",
    use_llmlingua2=True,
    device_map="cpu"  # runs on CPU, lightweight
)

# Compress a long document before sending to expensive API
compressed = compressor.compress_prompt(
    long_document,
    rate=0.33,           # keep ~33% of tokens
    force_tokens=['\n']  # never compress newlines
)

print(f"Original: {len(long_document.split())} tokens")
print(f"Compressed: {len(compressed['compressed_prompt'].split())} tokens")
print(f"Ratio: {compressed['ratio']:.1f}x")

Use case: RAG pipelines where retrieved context chunks are sent verbatim to the LLM. Compress the retrieved docs, keep the query intact. Typical result: 3x token reduction on context, minimal quality loss on factual Q&A.

2. Structured Output for Shorter Responses

Free-form responses are verbose. Models add preambles ("Certainly! Here's what I found..."), summaries ("In summary, the key points are..."), and filler. When you force structured output, you get exactly what you asked for.

from openai import OpenAI
from pydantic import BaseModel

class SentimentResult(BaseModel):
    sentiment: str  # "positive" | "negative" | "neutral"
    confidence: float
    key_phrases: list[str]

client = OpenAI()

# Unstructured: ~150 tokens of output for a sentiment analysis
# Structured: ~30 tokens of output for the same data
response = client.beta.chat.completions.parse(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": f"Analyze sentiment: {text}"}],
    response_format=SentimentResult
)

Switching from unstructured to JSON schema output on classification tasks typically reduces output tokens by 60–80%. On a high-volume classification system making 1M calls/month, this easily saves $500–2,000/month.

3. Request Batching

Most providers support batch APIs that process requests asynchronously at reduced pricing. OpenAI's Batch API gives a 50% discount. Anthropic's Message Batches API gives a 50% discount on processing. The trade-off: latency increases from milliseconds to minutes or hours.

from anthropic import Anthropic

client = Anthropic()

# Create a batch instead of individual requests
batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": f"doc_{i}",
            "params": {
                "model": "claude-haiku-3-5",
                "max_tokens": 300,
                "messages": [{"role": "user", "content": f"Summarize: {doc}"}]
            }
        }
        for i, doc in enumerate(documents)
    ]
)

print(f"Batch ID: {batch.id}")
# Poll later for results

Batching suits: nightly data processing, bulk document analysis, offline report generation, any workflow where you don't need results in real-time. Batch + cheap model is often the most cost-effective combination available.

4. Speculative Decoding

Speculative decoding uses a small "draft" model to generate candidate tokens, then verifies them in parallel using the target model. When the draft model's predictions are correct (often 70–80% of the time), you generate multiple tokens per forward pass of the big model.

This is a server-side optimization you get automatically from: - Groq's inference infrastructure - vLLM with speculative decoding enabled - Some OpenAI model deployments

For self-hosted inference with vLLM:

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-70B-Instruct \
    --speculative-model meta-llama/Llama-3.2-1B-Instruct \
    --num-speculative-tokens 5 \
    --use-v2-block-manager

Speculative decoding doesn't reduce token counts — it increases throughput on the same hardware. For self-hosted inference, this means serving 1.5–2.5x more requests with the same GPU, effectively halving per-request compute cost.

5. Quantization for Self-Hosted Models

Running models in FP16 (16-bit float) is the default. FP8 and INT4 quantization reduce memory footprint and compute requirements with acceptable quality trade-offs.

FP8 on vLLM is the production sweet spot for 70B+ models — you can fit a 70B model on a single A100 80GB instead of two, cutting hardware costs in half with under 2% quality degradation.

# vLLM with FP8 quantization
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-70B-Instruct \
    --quantization fp8 \
    --max-model-len 32768

For cloud inference, quantization is mostly abstracted away — providers do this themselves. The cost benefit is primarily for self-hosted setups.

6. Response Caching for Repeated Queries

Not all LLM calls are unique. In many production systems, 20–40% of queries are semantically identical or near-identical. Caching responses eliminates repeated inference entirely.

Exact match caching (Redis/Memcached) for truly identical prompts:

import hashlib
import redis

cache = redis.Redis()

def cached_completion(prompt: str, model: str, **kwargs) -> str:
    cache_key = hashlib.sha256(f"{model}:{prompt}".encode()).hexdigest()

    cached = cache.get(cache_key)
    if cached:
        return cached.decode()

    response = call_llm(prompt, model, **kwargs)
    cache.setex(cache_key, 3600, response)  # 1hr TTL
    return response

Semantic caching (GPTCache, Redis with vector search) for similar but not identical queries. A question like "What's your return policy?" and "Do you accept returns?" should hit the same cached response.

GPTCache wraps your existing LLM calls:

from gptcache import cache
from gptcache.adapter import openai

cache.init()  # defaults to SQLite + faiss vector index

# Now use gptcache's openai drop-in
response = openai.ChatCompletion.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": user_query}]
)

Semantic caching typically achieves 30–60% cache hit rates on FAQ/support systems. At $0.005 per request, 50% hit rate on 500K monthly requests = $1,250/month savings.

7. Output Length Control

The cheapest output token is one you never generate. Three mechanisms:

Explicit max_tokens: Set it per endpoint, not globally. A classification task that returns one word doesn't need max_tokens=4096.

Stop sequences: Tell the model to stop generating after a specific token or string. Useful for structured outputs where you know when the response is complete.

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[...],
    max_tokens=500,
    stop=["###END###", "\n\n\n"]  # halt on these tokens
)

"Brief" in system prompt: Adding "Be concise. Respond in 1–3 sentences unless more detail is explicitly requested." to your system prompt reduces output length by 30–50% on typical tasks. Simple, free, and consistently underused.

Combined, these three controls reduce output costs by 40–60% on most production endpoints without any architectural changes. It's the first thing to try before touching infrastructure.