Monitoring LLMs in Production

This cover signals the shift from mechanics to working code. The framing distinguishes understanding LLM monitoring from actually wiring up the tracing, scoring, drift detection, and alerting that make it real — which is where the engineering time goes and where most write-ups stop short.

Post four is deliberately code-heavy. The plan is laid out: a decorator that traces every call with latency and cost, a sampling-based judge pipeline, a nightly PSI drift job, a SQL query that powers a dashboard, and an alert rule that pages on a quality drop. Everything here is sized to drop straight into an existing service and adapt — these are patterns to steal, not a framework to adopt.

The first block is a tracing decorator, the foundation the rest builds on. It wraps any function that makes an LLM call, generating a unique trace ID and recording wall-clock latency around the call, then capturing the token counts and the output text the response carries. It emits that structured record to a sink and returns the original response untouched.

Two choices matter. Using a decorator means you instrument every call site by adding one line, rather than scattering logging logic through your code — instrumentation stays orthogonal to business logic. And the comment on emit flags that shipping should be asynchronous: the trace must never block or slow the user's response, a point the tips slide reinforces. This is the minimal, non-invasive way to get complete traces across a codebase.

The second block adds cost to the trace, turning raw token counts into dollars. It defines a pricing table keyed by model with separate input and output rates, computes a request's cost by multiplying token counts by the per-1K price, and folds that cost into the log record before emitting it.

The details reflect real practice. Input and output tokens are priced differently by every major provider, so the table separates them. Keying by model name means a single function handles a fleet of models correctly, which matters once you route cheap requests to a small model and hard ones to a large one. And the emit comment names where structured logs typically go — a message queue, a database, or an OpenTelemetry collector. Attaching cost at trace time is what makes the per-request budget guards and cost alerts elsewhere in the series possible.

The third block is the sampled judge scorer, which implements the two-tier sampling strategy from the mechanics post in a few lines. It always scores the suspicious cases — anything with a thumbs-down or an unusually high cost — and otherwise scores a small random percentage of traffic. For anything selected, it calls the judge, saves the score and reason against the trace ID, and raises an alert if the score is low.

The logic encodes the cost-versus-coverage tradeoff directly. The forced conditions guarantee you never miss a known-bad request, while the random sample gives you an unbiased estimate of overall quality without paying to grade everything. Saving the reason alongside the score preserves auditability, and alerting only on genuinely low scores keeps the signal actionable rather than noisy. This function is the heart of scalable quality measurement.

The fourth block is the nightly drift job, which ties the PSI implementation from the mechanics post into a scheduled comparison. It pulls judge scores from a baseline window — here the week running from fourteen to seven days ago — and from the last day of live traffic, computes the PSI between the two distributions, records it as a metric, and alerts if it crosses the 0.2 threshold.

The design choices are practical. Comparing a recent live window against a slightly older baseline window catches gradual drift without being fooled by normal daily noise. Recording the PSI as a metric means you can chart drift over time, not just alert on it, so you see a trend building before it crosses the line. Running nightly is a sensible cadence for the slow degradation that drift represents — you don't need second-by-second drift detection, you need a reliable daily check.

The fifth block is the dashboard query, which shows that the core health view is plain SQL over a traces table. It buckets the last 24 hours of traces by hour and computes the four numbers that summarize LLM health: request volume, p95 latency using a percentile aggregate, total spend, and average quality score.

The query is deliberately readable because the lesson is that you don't need a specialized observability vendor to get a useful dashboard — a well-structured traces table and one grouped query give you the four pillars at a glance. Using p95 rather than average latency is intentional, since tail latency is what users actually feel. Pulling volume, cost, and quality into the same time series lets you spot correlated movements — a quality drop that coincides with a cost spike, for instance — which is exactly the kind of pattern a single combined view reveals and separate dashboards hide.

The pipeline diagram zooms back out to show how the code blocks connect into one system. The tracing decorator captures every request. Traces land in a storage table. A sampled judge scores a slice of them. And drift detection plus live alerts watch for trouble — drift on a nightly cadence, quality and cost in near real time.

The diagram's job is to keep the reader oriented amid the code. Each block they just read is one stage here, and the arrows show the data flow: capture, store, score, watch. Seeing the whole shape makes it clear that none of the pieces is useful alone — tracing without scoring is just logs, scoring without alerting is just numbers. It's the assembled pipeline that delivers the always-on loop post one promised.

The sixth block is an alert rule, expressed in a Prometheus-style configuration that most ops teams will recognize. It fires a paging alert when the average judge score over the last hour stays below 3.5 for a sustained 15 minutes, and it annotates the alert with the offending value so the on-call engineer has context immediately.

Two design decisions directly counter the alert-fatigue mistake the final post details. The 'for: 15m' clause means a single bad response or a brief blip won't page anyone — the condition has to hold, so you alert on a sustained regression, not noise. And the severity label routes it as a page precisely because a sustained quality drop is the kind of thing that needs a human now. Alerting on a quality metric, not just on errors or CPU, is the whole point: this is the alert traditional monitoring can't express.

The tips slide gives the production rules that tie the code together. Emit traces asynchronously so monitoring never slows the user-facing response. Sample to control cost but force-score the suspicious cases so you never miss a known failure. Store the raw prompt and retrieved context so any request can be replayed during debugging. Alert on quality and cost, not just errors, because those are the failures unique to LLMs. And version your prompts so that when quality moves, you can attribute the change to a specific edit.

These five rules are what separate a monitoring setup that works at scale from one that demos well and falls over in production. Several are the positive form of mistakes the final post catalogs, and prompt versioning in particular is what makes the 'a prompt tweak is a deploy' risk from post two actually debuggable.

The closing mistake targets a serious and common failure: logging PII into your traces. The whole value of tracing is capturing the raw prompt and response for replay — but raw user prompts are full of names, emails, and account numbers, and dumping them into logs the whole company can query turns your debugging tool into a breach waiting to be reported.

The fix is to redact or hash PII before it reaches the sink, set retention limits so sensitive data doesn't accumulate forever, and gate who can read traces. The tension is real — you want enough of the prompt to debug, without storing identifiers you'd have to disclose in an incident. The final post provides a concrete redaction function. Treating trace data as sensitive from the start is far cheaper than discovering, after a breach, that your monitoring system was the liability.

The CTA hands off to the final post, which catalogs the common mistakes that make LLM monitoring noisy, blind, or worse than nothing — watching infra instead of output, alert fatigue, trusting an uncalibrated judge, ignoring cost, logging PII, and collecting metrics nobody acts on. The teaser frames it as the failure manual that completes the toolkit.