7 Smart Techniques to Slash LLM Costs in Production

Prices and model names in this article were updated on 31 October 2026.

Switching to a cheaper model is the obvious move; see model routing for how to do it per request. After that, the seven techniques below cut what each request costs. Most sections include a worked example at October 2026 list prices. Redo the arithmetic with your own volumes.

1. Prompt compression with LLMLingua

LLMLingua is a Microsoft open-source project that removes low-information tokens from a prompt using a small language model. The project says it achieves "up to 20x compression with minimal performance loss". It also says LLMLingua-2 is 3x to 6x faster than the original. The latest release on PyPI is 0.2.2, from April 2024. Measure the quality loss on your own task before trusting either figure.

from llmlingua import PromptCompressor

compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
    use_llmlingua2=True,
)

# Compress retrieved context before sending it to the paid API
compressed = compressor.compress_prompt(
    retrieved_context,
    rate=0.33,                 # keep about a third of the tokens
    force_tokens=['\n', '?'],  # never drop these
)

print(compressed["compressed_prompt"])

Use it on retrieved context in RAG pipelines, where chunks go to the LLM verbatim. Compress the chunks and leave the question intact.

A worked example: 1M RAG calls a month on Claude Sonnet 5.5 ($2 per million input tokens), each with 6,000 tokens of retrieved context. The context costs 6B × $2 / 1M = $12,000 a month. At rate=0.33 it shrinks to about 2,000 tokens, or about $4,000. The compressor runs on your own hardware.

Don't compress a prefix you already cache. On Sonnet 5.5 a cached read costs $0.10 per million tokens, so compression saves little there. It also changes the prefix from request to request, which breaks the cache. See the prompt caching guide.

2. Structured output for shorter responses

Free-form answers run long. Models add preambles ("Certainly! Here's what I found...") and closing summaries. Structured output returns only the fields you asked for.

from openai import OpenAI
from pydantic import BaseModel

class SentimentResult(BaseModel):
    sentiment: str  # "positive" | "negative" | "neutral"
    confidence: float
    key_phrases: list[str]

client = OpenAI()

response = client.chat.completions.parse(
    model="gpt-6-astra",
    messages=[{"role": "user", "content": f"Analyze sentiment: {text}"}],
    response_format=SentimentResult
)
result = response.choices[0].message.parsed

What it saves depends on the output price. Suppose a free-form answer averages 150 output tokens and the JSON version 30; measure your own. Over 1M calls a month that removes 120M output tokens. On GPT-6 Luna ($0.50 per million output tokens) the saving is $60 a month. On GPT-6 Astra ($50 per million) it is $6,000.

3. Request batching

The large vendors sell asynchronous batch processing at a discount. You upload a file of requests and collect the results later.

from anthropic import Anthropic

client = Anthropic()

# One batch instead of individual requests
batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": f"doc_{i}",
            "params": {
                "model": "your-small-claude-model",
                "max_tokens": 300,
                "messages": [{"role": "user", "content": f"Summarize: {doc}"}]
            }
        }
        for i, doc in enumerate(documents)
    ]
)

print(f"Batch ID: {batch.id}")
# Poll later for results

For your-small-claude-model, use Claude Haiku 5.5 (take the exact model id from Anthropic's models page); Sonnet 5.5 and Opus 5.5 also work in batches.

Batching suits any job that can wait, such as nightly processing, bulk document analysis, offline reports and evaluations. A worked example: summarize 100,000 documents of 2,000 tokens each into 300 tokens. On Claude Opus 5.5 that is 200M input × $4 + 30M output × $20 = $1,400, or $700 as a batch. On Claude Haiku 5.5 it is 200M × $0.10 + 30M × $0.50 = $35, or $17.50 as a batch. A small model in a batch costs 1/80 of the flagship run synchronously. Details per vendor are in the batch API comparison.

4. Speculative decoding

Speculative decoding uses a small draft model to propose several tokens, then has the target model verify them in one pass. Accepted tokens arrive faster. vLLM's documentation describes the goal as reducing "inter-token latency under medium-to-low QPS (queries per second), memory-bound workloads". Its own table rates the draft-model method's gain lower at high request rates ("High gain" at low QPS, "Medium gain" at high QPS).

The output is the same, so the token count, and any per-token bill, is unchanged. The saving is in hardware. A self-hosted model can meet its latency target with fewer GPUs at low traffic. At high traffic the draft-model gain shrinks.

For self-hosted inference with vLLM, using its draft-model method (by default the draft and target models must share a vocabulary):

vllm serve <target-model> \
    --speculative-config '{"method": "draft_model", "model": "<draft-model>", "num_speculative_tokens": 5}'

vLLM also supports other proposal methods, including EAGLE, multi-token prediction (MTP) and n-gram matching.

5. Quantization for self-hosted models

Weights stored in 16-bit floats (FP16) take 2 bytes per parameter. FP8 takes 1 byte and INT4 half a byte. The memory for the weights follows directly:

The KV cache and activations need memory on top of this. A 70B model in FP16 needs two 80 GB GPUs for the weights alone. At FP8 the weights fit on one 80 GB card. That leaves about 10 GB for the KV cache, which limits context length and concurrency. Quality loss differs by model and task, so measure it on your own evaluation set.

# vLLM with online dynamic FP8 quantization
vllm serve meta-llama/Llama-3.3-70B-Instruct \
    --quantization fp8_per_tensor \
    --max-model-len 32768

Some published reference points follow. The AWS model card says gpt-oss-20b fits in 16 GB of memory. TestingCatalog reported that gpt-oss-120b runs on one 80 GB GPU. Ollama's README suggests at least 8 GB of RAM for 7B models, 16 GB for 13B and 32 GB for 33B. According to Pulse, Cohere's Command A+ runs in 4-bit on one B200 or two H100s.

For hosted APIs, quantization is the provider's business; the saving reaches you only through their price list. For small models on your own hardware, see small language models.

6. Response caching for repeated queries

Some LLM calls repeat. Caching the response skips the call entirely.

Exact-match caching (Redis, Memcached) for identical prompts:

import hashlib
import redis

cache = redis.Redis()

def cached_completion(prompt: str, model: str, **kwargs) -> str:
    cache_key = hashlib.sha256(f"{model}:{prompt}".encode()).hexdigest()

    cached = cache.get(cache_key)
    if cached:
        return cached.decode()

    response = call_llm(prompt, model, **kwargs)
    cache.setex(cache_key, 3600, response)  # 1-hour TTL
    return response

Semantic caching matches questions that are similar but not identical, such as "What's your return policy?" and "Do you accept returns?". It embeds each query and returns a stored answer when a new query is close enough. Gateways offer it as a setting. LiteLLM's proxy lists redis-semantic and qdrant-semantic cache types. Portkey lists semantic caching on its Pro and Enterprise plans. The risk is serving the stored answer to a question that differs in a detail that matters. Set the similarity threshold conservatively and keep personalized answers out of the cache.

A worked example: a support bot on Claude Sonnet 5.5 with 3,000 input and 300 output tokens per request. One request costs 3,000 × $2 / 1M + 300 × $10 / 1M = $0.009. At 500,000 requests a month the bill is $4,500. Every 10% of requests served from cache saves $450 a month. Your logs tell you which share repeats.

7. Output length control

The cheapest output token is the one you never generate. Output costs five times input on most current price lists. Claude Sonnet 5.5 charges $2 / $10 and GPT-6 Astra $10 / $50. Three controls apply.

max_tokens per endpoint

A classification task that returns one word does not need max_tokens=4096. Give each endpoint its own limit instead of one global value.

Stop sequences

The model stops when it produces a given string. This helps with structured outputs where you know where the answer ends.

A brevity instruction

Add one line to the system prompt, for example "Be concise. Respond in 1–3 sentences unless more detail is requested." It costs nothing to try.

from anthropic import Anthropic

client = Anthropic()

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=500,
    stop_sequences=["###END###"],
    system="Be concise. Respond in 1-3 sentences unless more detail is requested.",
    messages=[{"role": "user", "content": question}]
)

Reasoning counts as output too. Anthropic's Claude Code documentation states that "Thinking tokens are billed as output tokens". It adds that the default budget "can be tens of thousands of tokens per request depending on the model".

A worked example: trimming 100 output tokens per answer on 1M Sonnet 5.5 calls a month removes 100M output tokens, or $1,000 a month.

What to do next

• Start with output length and response caching, since both work on any hosted API and need no new infrastructure.
• Move every job that can wait a day to a batch API.
• Try compression and quantization only with an evaluation set in place, since both trade quality for cost.
• We can rank these for a live pipeline as part of an LLM cost audit.

Sources

• LLMLingua on PyPI
• Claude pricing (platform.claude.com)
• OpenAI models overview
• OpenAI Batch API guide
• Anthropic Message Batches
• Gemini Batch API
• vLLM: speculative decoding
• vLLM: FP8 quantization
• AWS Bedrock model card: gpt-oss-20b
• LiteLLM: proxy caching
• Portkey: plan feature comparison