Observability for LLM Applications

An LLM feature can fail without throwing a single error. The API returns 200, latency looks normal, and the answers are quietly wrong, or twice as expensive as last week. Traditional monitoring catches outages. LLM applications also need visibility into cost and quality. Here is what we instrument and why.

The basics still apply

Start with what you would monitor for any service: request rate, error rate, latency percentiles and saturation. For self-hosted models add GPU utilisation, GPU memory, queue depth and KV cache usage. vLLM and TGI expose Prometheus metrics for most of these, and the NVIDIA DCGM exporter covers the hardware.

LLM specific metrics

On top of the basics, record these for every model call:

Metric	Why it matters
Input and output tokens	Drives cost and latency; spikes reveal prompt bloat or runaway outputs
Time to first token	What users feel in a streaming interface
Tokens per second	Throughput of the model or provider
Total latency	End to end, including retrieval and tool calls
Cost per request	Tokens multiplied by price, attributed to feature, team or customer
Finish reason	Length cut-offs and content filter stops signal problems
Retries and fallbacks	Hidden cost and a sign of provider trouble

Label metrics by model, prompt version and feature. When cost jumps, you want to know which change caused it without guesswork.

Traces show the whole request

A single user request in an LLM application can involve retrieval, reranking, several model calls and tool invocations. Distributed tracing makes that chain visible.

OpenTelemetry now has semantic conventions for generative AI (the gen_ai.* attributes) covering model name, token usage and operation type. Instrumentation libraries such as OpenLLMetry and OpenLIT, as well as many frameworks, emit these spans. Send them to whatever tracing backend you already use: Grafana Tempo, Jaeger, Honeycomb, Datadog or others. Tools built for LLMs, such as Langfuse and Arize Phoenix, accept OpenTelemetry data and add prompt and evaluation views on top.

Keeping to OpenTelemetry means LLM traces sit next to the rest of your system’s traces, so a slow database query and a slow model call appear in the same view.

Be careful with prompt content

Prompts and responses are the most useful debugging data and the most sensitive. They can contain personal data, customer documents and secrets. Decide deliberately:

• Which environments store full prompt and response text.
• How long it is retained, and who can see it.
• Whether personal data is redacted before storage.

Often the right answer is metrics and metadata everywhere, full content only for a sampled subset with restricted access.

Quality signals

Quality is the hardest part to observe, but there are useful signals:

1. User feedback. Thumbs up and down, edits to generated text, or a user retrying the same question.
2. Automatic checks. Does output parse as the expected JSON? Does it cite a source? Is it within length limits?
3. LLM-as-judge scoring. A second model grades a sample of responses for relevance or faithfulness against the retrieved context. Calibrate it against human judgement before trusting it.
4. Offline evaluation. Your test set, rerun on a schedule and on every change, with results tracked over time.

Plot these next to prompt and model versions. A drop that lines up with a deploy is easy to diagnose; a gradual decline usually points to drifting input data.

Alerts worth having

• Cost per hour or per day above an expected band.
• Error or fallback rate rising for one provider or model.
• Time to first token above your target at p95.
• Share of failed structured output parses above a threshold.
• Evaluation score below the last accepted baseline.

Keep it in one place

Avoid a separate island for AI monitoring. Prometheus, Grafana and an OpenTelemetry Collector can carry all of the above, alongside the dashboards your team already uses for the rest of the system.

DoneDeploy sets up monitoring and tracing for LLM applications using the same open tooling we use for the rest of your infrastructure. If you are flying blind on cost or quality, get in touch.