LLM Observability

The practice of continuously measuring quality, safety, cost, and performance of LLM outputs in production. Differs from traditional APM because LLMs are non…

LLM Observability

The practice of continuously measuring quality, safety, cost, and performance of LLM outputs in production. Differs from traditional APM because LLMs are non-deterministic — the same input can produce different outputs — and failure modes are semantic rather than structural (HTTP 200 masks hallucinations, wrong tool choices, silent reasoning failures).

Why LLM Observability Differs From APM

Signal	Traditional app	LLM / AI agent
Latency driver	CPU, I/O, network	Token count, model size, context window
Cost unit	Requests/second	Tokens consumed
Failure mode	Exception, timeout	Hallucination, context overflow, tool error
Debug artifact	Stack trace	Prompt + completion + reasoning chain

Even at temperature=0, a large frontier model produced 80 unique completions across 1,000 identical runs (training data — verify). Traditional dashboards show green while users report failures.

Three Pillars: Traces, Metrics, Logs

Traces — end-to-end request flows. For agents, a trace covers the full reasoning chain: agent root span → LLM call spans → tool call spans → retrieval spans. The trace_id propagates across service boundaries via W3C traceparent headers.

Metrics — aggregate time-series. Token burn rate, p99 latency, error rate. GenAI conventions define standard metric names: gen_ai.client.token.usage (counter), gen_ai.client.operation.duration (histogram), gen_ai.server.time_to_first_token (histogram for streaming).

Logs / Events — structured records for prompt/completion content, audit trails, compliance. Content lives in span events (not attributes) so it can be filtered at the Collector without code changes.

GenAI Semantic Conventions

The OTel community’s OpenTelemetry GenAI Special Interest Group defines standard gen_ai.* attribute names. Status: incubating but rapidly stabilizing. Using them ensures telemetry works with any OTel-compatible backend without custom parsing.

Core Span Attributes

Attribute	Description	Example
gen_ai.system	LLM provider	"anthropic", "openai", "google_vertex_ai"
gen_ai.operation.name	Operation type	"chat", "embeddings", "text_completion"
gen_ai.request.model	Requested model	"claude-opus-4-6", "gpt-4o"
gen_ai.response.model	Actual model used	may differ from requested
gen_ai.request.temperature	Sampling temp	0.7
gen_ai.request.max_tokens	Max tokens requested	1024
gen_ai.usage.input_tokens	Prompt tokens consumed	512
gen_ai.usage.output_tokens	Completion tokens	128
gen_ai.response.finish_reasons	Why generation stopped	["stop"], ["length"]

Span Events for Content

Prompt and completion content belongs in span events, not attributes. Attributes are always indexed and exported; events can be dropped or redacted at the Collector level.

Event name	When
gen_ai.system.message	Before LLM call
gen_ai.user.message	Before LLM call
gen_ai.assistant.message	After LLM call
gen_ai.tool.message	After tool result

Agent-Specific Attributes

gen_ai.agent.name, gen_ai.agent.id — added to keep each decision node traceable in multi-agent orchestration. gen_ai.tool.name for MCP/tool call identification.

Span Types in an Agent Trace

A properly instrumented agent trace hierarchy:

agent.run (root span)
├── gen_ai.embeddings          ← embedding call
├── db.vector_search           ← retrieval step
├── gen_ai.chat                ← LLM call, decides to use tools
│   ├── agent.tool_call: web_search
│   └── agent.tool_call: calculator
└── gen_ai.chat                ← LLM call, final answer

For MCP workflows specifically:

Agent Run (root)
├── MCP Discovery
│   └── mcp.tools.count: 5
├── Agent Planning
│   └── Selected tool: read_file
└── MCP Tool Execution
    ├── gen_ai.operation.name: execute_tool
    ├── gen_ai.tool.name: read_file
    └── Duration: 340ms

Each MCP tool call creates its own child span. Context propagates across MCP server boundaries the same way it does across HTTP services.

Metrics: What to Track

Six essential KPIs for agent systems:

1. Token usage per run — input + output tokens per operation (cost signal)
2. Tool call success rate — % successful MCP/tool invocations
3. LLM latency distribution — p99 especially; often the SLA metric
4. Agent loop iterations — ReAct cycles before task completion
5. Context window utilization — % of available context consumed
6. End-to-end agent latency — user request to final response

Alert thresholds (from sources):

• gen_ai.client.token.usage rate > 2× baseline over 10 min → runaway loop or prompt injection
• gen_ai.client.operation.duration p99 > 30s → model overloaded or context too large
• Error rate > 2% over 5 min → rate limiting or quota exhaustion
• Input/output token ratio > 10:1 consistently → system prompt too large

RED Method for Agents

Rate, Errors, Duration applied to agent workflows:

• Rate: tool invocations per second, LLM calls per agent run
• Errors: tool execution failures, LLM errors, policy violations
• Duration: per-tool latency, per-LLM-call latency, total agent runtime

Quality and Safety Metrics

Beyond infrastructure signals, agent observability needs semantic metrics:

Quality: groundedness/faithfulness, answer relevancy, context precision, coherence. Treat as operational signals tied to SLAs, not abstract research scores.

Safety: hallucination rate (production teams target < 0.5%), toxicity score, prompt injection detection rate, PII leakage rate. Safety metrics should not live in a separate dashboard — a prompt injection can change tool choice, trigger bad actions, and cause a cost anomaly in the same session.

Drift: Jensen-Shannon Distance and PSI between production inputs and reference baselines. Drifting prompts erode margins before anyone flags an issue.

Cost Tracking

Cost attribution at request and prompt-version level prevents silent spend explosions. A prompt tweak that appends verbose text can double daily token usage within a week. Track:

• Cost per request (gen_ai.usage.cost or derived from token counts × model pricing)
• Cost per team/project/tool (requires tagging at span level)
• Token efficiency: which prompts generate wasteful output tokens

Logging: Structured and Sampled

For compliance and audit:

• gen_ai.agent.id, gen_ai.tool.name, gen_ai.request.model — log metadata, not content
• Log input/output sizes, not content, to protect PII
• Correlate audit logs with trace IDs for incident reconstruction

PII protection: set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false to capture token counts without prompt/response text. For partial visibility, sample content at 10% or truncate to first 500 chars.

Sampling Strategies

100% tracing at production LLM volume is expensive. Recommended:

Scenario	Strategy	Rate
Development	AlwaysOn	100%
Production, successful calls	TraceIdRatioBased	5–10%
Production, errors	Tail-based	100%
High-token requests (> 2K tokens)	Tail-based attribute filter	100%
Agent runs	Tail-based	100% (rare + high-value)

Head sampling — decision at trace start, based on trace ID. Simple, low overhead, but can’t consider what happened in the request. Wrong tradeoff for AI: drops 90% of errors if set to 10%.

Tail sampling — decision after trace completes. Buffer all spans, then decide: keep errors, keep slow traces, keep enterprise users, sample 5% of the rest. Higher memory cost but captures what matters.

Multi-Agent Trace Correlation

In multi-agent systems, trace_id propagates through all service boundaries. W3C traceparent header carries {version}-{trace_id}-{parent_span_id}-{flags} across HTTP calls automatically when HTTP clients are instrumented.

For message queues (Celery, Kafka, SQS): context does NOT propagate automatically. Must manually inject(headers) when producing messages and extract(headers) when consuming. Missing this creates disconnected root spans — different requests appear in the same trace.

Agent-specific challenges:

• Non-linear execution: tool call chains branch and fan out; traces are trees not lines
• Async traces: streaming responses, async tool calls, background agents
• Thread pools: worker threads don’t inherit OTel context — must capture and pass current_context explicitly, always detach in finally block

Key Tools and Platforms

Auto-instrumentation libraries (instrument 40+ frameworks without code changes):

• OpenLLMetry (traceloop-sdk) — LangChain, LangGraph, multi-language (Python/JS/Go/Ruby)
• OpenInference (Arize) — LlamaIndex, AutoGen; tight integration with Phoenix eval platform
• OpenLIT — broadest framework support (AG2, Dynamiq, Mem0); zero-code CLI option

Observability backends:

• Arize Phoenix — local dev analysis, UMAP visualization of embeddings, eval integration
• LangSmith / Langfuse — AI-specific observability, prompt versioning, dataset curation
• Grafana (+ Tempo) — open-source, self-hosted; GenAI dashboards via OpenLIT SDK
• Datadog / Honeycomb — enterprise APM with OTel GenAI support
• AgentOps — agent-specific monitoring (training data — verify current status)
• Galileo — agent observability + guardrails platform with purpose-built eval models

OTel Collector — standalone process that receives spans, applies sampling/enrichment, routes to multiple backends. Two deployment modes:

• Agent mode: sidecar per application instance, low latency
• Gateway mode: centralized tier for sophisticated routing, preferred at scale

Production Patterns

Drift detection: monitor production prompt distributions against baseline using Jensen-Shannon Distance. Prompt drift erodes quality before metrics catch it.

Eval-to-guardrail lifecycle: pre-production eval logic (quality gates, safety checks) can be deployed as runtime guardrails post-launch. Same standards apply across both phases.

Runtime guardrails: intercept risky outputs before they reach users. Architecture: rules triggered on metric values → rulesets evaluated in parallel → stages with OR logic → actions (override, redact, webhook). Creates audit trail linked to session trace.

Cost-aware eval: purpose-built small judge models (8B–14B params) achieve comparable eval quality at 46–82% lower cost than GPT-4o-based eval (training data — verify). Enables scoring every conversation rather than sampling.

Failure pattern clustering: anomaly detection across production traces surfaces “unknown unknowns” — infinite loops, stalled progress, cascading failures — that manual search misses.

Security Considerations

Telemetry pipelines handle sensitive data. Essential controls:

• TLS 1.2+ for all OTLP export connections
• PII redaction at instrumentation level or Collector transform processor
• RBAC at observability backend level
• Data residency: configure Collectors to route telemetry to region-specific storage
• Anomaly detection on telemetry: token consumption spikes → potential prompt injection; unusual tool invocation patterns → potential compromise

Anti-Patterns

• Storing prompt content in span attributes (always indexed, always exported, creates PII risk and cost) — use span events instead
• Head-only sampling (misses 90% of errors) — use tail-based for production
• Single “total tokens” counter (hides cost structure) — track input and output separately
• Custom attribute names instead of gen_ai.* conventions (breaks pre-built dashboards)
• Not instrumenting HTTP clients in distributed pipelines (context breaks at service boundaries)
• Not flushing BatchSpanProcessor in serverless/Lambda functions (silently drops traces)

Related Pages

• OpenTelemetry — the framework providing the instrumentation layer
• Context Degradation Patterns — failure modes that observability helps diagnose
• Agent Harness — orchestration patterns where trace correlation matters