You can't optimize what you can't measure. Most teams building with LLMs start by looking at their API provider's billing page — which tells you total spend but nothing about which feature, which user, which prompt, or which model tier is responsible for it.

LLM observability tools fill that gap. Here's what's worth using and what metrics actually matter.

The Tooling Landscape

Helicone ($20/month and up)

Helicone is a proxy-based observability tool — your requests pass through Helicone's servers, which log everything before forwarding. Zero code changes needed for basic setup.

from openai import OpenAI

# Replace the base URL — that's it
client = OpenAI(
    api_key="your-openai-key",
    base_url="https://oai.helicone.ai/v1",
    default_headers={"Helicone-Auth": f"Bearer {helicone_key}"}
)

# Add metadata for better cost attribution
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[...],
    extra_headers={
        "Helicone-User-Id": user_id,
        "Helicone-Property-Feature": "report-generation",
        "Helicone-Property-Team": "analytics"
    }
)

Helicone's dashboard breaks down costs by user, feature, model, and time period. You can set rate limits per user, run A/B experiments on prompts, and get alerts when spend exceeds thresholds. The free tier handles 10,000 requests/month.

The downside: all your prompts and responses pass through their servers. Not suitable for sensitive data without reviewing their data processing terms.

LangSmith (Free tier, $39/month for teams)

LangSmith is Langchain's observability platform. It's SDK-based rather than proxy-based — you instrument your code with LangSmith's tracing decorator, which sends trace data to their servers.

from langsmith import traceable
from langsmith.wrappers import wrap_openai
from openai import OpenAI

client = wrap_openai(OpenAI())

@traceable(name="support-response", metadata={"feature": "support", "tier": "pro"})
def generate_support_reply(ticket: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": ticket}]
    )
    return response.choices[0].message.content

LangSmith is strongest for multi-step chains and agents — it shows you the full trace of each agent run, including which steps were slow and which were expensive. For a single-call application, it's overkill. For complex agent workflows, it's invaluable.

PromptLayer

PromptLayer is simpler than LangSmith — focused on prompt versioning and cost tracking rather than full workflow tracing. The core value is comparing prompt versions against each other on cost and quality.

import promptlayer

promptlayer.api_key = "your-pl-key"
openai = promptlayer.openai  # drop-in replacement

response = openai.ChatCompletion.create(
    model="gpt-4o",
    messages=[...],
    pl_tags=["production", "support-v2"]  # tag for tracking
)

Use PromptLayer if your primary concern is prompt lifecycle management — tracking which version of a prompt is live, what it costs, and how it performs. Not the right tool if you need agent tracing or real-time alerting.

Portkey (free tier, $49/month+)

Portkey combines proxy routing with observability. Covered in the load balancers article, but from a cost-tracking perspective: Portkey's dashboards show cost per user, per feature, and per model in real time. It's the fastest path to cost attribution without building custom logging infrastructure.

Phoenix by Arize (Open Source)

Phoenix is Arize AI's open-source LLM tracing and evaluation platform. It runs locally (or self-hosted) and requires no data to leave your infrastructure — critical for sensitive workloads.

pip install arize-phoenix openinference-instrumentation-openai

# Start Phoenix server locally
python -m phoenix.server.main &

# Instrument your app
from openinference.instrumentation.openai import OpenAIInstrumentor
from phoenix.otel import register

register(endpoint="http://localhost:6006/v1/traces")
OpenAIInstrumentor().instrument()

# All OpenAI calls are now traced automatically

Phoenix gives you traces, latency analysis, and cost breakdowns in a local UI. No SaaS, no data egress, no recurring cost. The trade-off: you manage the server, and the UI is less polished than Helicone.

OpenLLMetry (Open Source)

OpenLLMetry (by Traceloop) is an OpenTelemetry-based instrumentation layer for LLMs. It integrates with any OTel-compatible backend — Datadog, Grafana, Jaeger, etc.

from traceloop.sdk import Traceloop
from traceloop.sdk.decorators import task, workflow

Traceloop.init(app_name="your-app", api_key="your-key")

@workflow(name="document-processing")
def process_document(doc: str):
    summary = summarize(doc)
    entities = extract_entities(summary)
    return entities

@task(name="summarize")
def summarize(text: str) -> str:
    # your LLM call here
    ...

If you're already running OTel infrastructure, OpenLLMetry is the cleanest integration — LLM traces flow into the same Grafana dashboards as your other service metrics. Useful for engineering teams who want LLM cost data alongside application performance data.

What Metrics to Actually Track

Most teams set up these tools and then look at total cost. That's table stakes. The metrics that drive action:

Cost per session. Total spend divided by number of distinct user sessions. If this number is growing faster than your user count, your application is getting more expensive per user over time — usually because context windows are growing without truncation.

Cost per task type. Break down spend by feature (e.g., report-generation, support-chat, data-extraction). You almost always find one feature consuming a disproportionate share. Optimize that first.

Cache hit rate. If you've implemented prompt caching (KV cache with Anthropic, prefix cache with OpenAI), track the percentage of input tokens served from cache. Below 40% on a workload that should benefit from caching means your prompt structure needs fixing.

Model utilization by tier. What percentage of calls hit your frontier model vs. your economy model? Target: no more than 20–30% frontier for most applications.

P95 token count. The median request is fine. The P95 is where expensive outliers live — long context injections, runaway agent loops, unusually verbose responses. Set alerts on P95 token counts exceeding 2–3x the median.

Cost per successful outcome. This requires defining what "success" means for your application, but it's the metric that matters most. A 10% quality improvement that doubles cost per outcome is a bad trade. A 20% quality improvement that adds 5% to cost is a good one.

Quick-Start Recommendation by Team Size

Solo developer or small startup: Helicone free tier. Zero setup friction, immediate cost visibility, upgrade to paid when you need user-level attribution.

Mid-size team with complex agents: LangSmith team plan. The agent tracing is worth the cost once you're debugging multi-step workflows.

Enterprise with data privacy requirements: Phoenix (self-hosted) + OpenLLMetry feeding into existing Grafana/Datadog infrastructure. No data leaves your network.

Team already using LiteLLM proxy: Use LiteLLM's built-in logging to PostgreSQL and connect a BI tool (Metabase, Grafana). Free, and the data model is straightforward enough to query directly.

Don't over-instrument early. Pick one tool, get it shipping data, and act on what you learn before adding more complexity.