Monitoring LLMs in Production

This cover sets the agenda for the mechanics post by leading with the scale problem that forces the whole design: you cannot put a human reviewer on tens of thousands of daily responses, so how do you actually know your quality? The honest answer is a system of four interlocking parts, and naming that scale constraint up front motivates every technique that follows.

Post three is the engine room. We've established what LLM monitoring is and why it matters; now we explain the actual machinery — tracing, sampling, automated judging, reference-free metrics, drift detection, and the user-feedback loop — so that running quality at scale stops being guesswork and becomes a system you can reason about and build.

The foundation is tracing every request end to end. A trace captures one request's full journey: the user's input, the context retrieved for RAG, each tool call the system made, the final assembled prompt actually sent to the model, the raw response that came back, and timing for every step. Spans nest under a single trace ID so the whole thing can be replayed exactly as it happened.

The reason tracing comes first is that it's the raw material everything else depends on. Scoring needs the prompt and response; debugging needs the retrieved context; drift detection needs the input features. Without complete traces you're monitoring a black box. With them, any single request becomes reconstructable after the fact — which is the difference between knowing something broke and knowing why.

The cycle diagram shows the monitoring loop as a continuous process rather than a one-shot measurement. You trace each request to log the full interaction. You sample a slice of those traces rather than processing all of them. You score the sample with a judge and with cheap metrics. You detect drift and fire alerts by comparing distributions over time. And you act — fixing a prompt, rolling back a model, refreshing a stale index — which changes the system and feeds the next round of traces.

Presenting it as a cycle makes the key point that monitoring isn't a dashboard you build once; it's a loop that keeps running and keeps feeding decisions. The 'act' node is what distinguishes monitoring that pays off from monitoring that's merely decorative, a theme the final post returns to.

This slide explains sampling and why it's necessary rather than lazy. Running an LLM judge on 100% of traffic means paying for inference twice — once to serve the user and once to grade the answer — and it adds latency and load you don't need. So you score a random sample, perhaps 1 to 5 percent, which is enough to track quality trends across the whole population with statistical confidence.

But random sampling alone would miss rare, important failures, so you layer on targeted sampling: always score low-confidence outputs, anything a user gave a thumbs-down, and unusually high-cost requests. The result is a microscope pointed at the cases most likely to be broken, combined with a representative random slice for trend tracking. This two-tier strategy is how you get both coverage and focus without scoring everything.

This slide introduces LLM-as-a-judge, the technique that makes automated quality measurement possible when you have no ground-truth labels — which is almost always the case in production. You give a strong model a rubric and ask it to grade an output: is this relevant, is it grounded in the provided context, is the format correct? The judge returns a numeric score and a written reason.

The honest framing matters: judges are not perfect. They carry biases, which the final post catalogs in detail. But calibrated against human ratings, an LLM judge scales quality measurement to volumes no human team could ever review. The reason it returns is as valuable as the score, because it turns a number into something a human can audit and a runbook can act on. This is the core of modern automated LLM evaluation.

This code slide turns LLM-as-a-judge into a concrete, runnable call. It defines a rubric that asks specifically for faithfulness — whether the answer is supported by the context — and demands a structured JSON response with a 1-to-5 score and a reason. It then sends the context and answer to a strong model with two important settings: a forced JSON response format so the output parses reliably, and temperature 0 so the grading is as deterministic as possible.

The details encode best practice. Asking for a reason, not just a score, gives you auditability. Forcing JSON output prevents the parsing failures that plague free-text judges. Temperature 0 reduces run-to-run variance so the same answer gets a stable grade. This faithfulness check is the single most useful judge to start with for any RAG system, because hallucination against retrieved context is the most common quality failure.

This slide covers reference-free metrics — quality signals that need no labels and no judge call at all, which makes them cheap enough to run on every single request. Groundedness can be approximated by checking whether the claims in an answer actually appear in the retrieved context. Format validity is just parsing the output against an expected schema and recording pass or fail. Refusal rate and empty-response rate are simple rule-based counts.

The strategic value is layering. These deterministic checks catch whole categories of failure — malformed JSON, empty answers, obvious ungroundedness — instantly and for free, before you ever spend money on a judge call. A well-designed monitoring system runs cheap reference-free checks on everything and reserves the expensive judge for the sampled slice, which is the most cost-effective way to cover both breadth and depth.

The flow diagram lays out drift detection as a four-step comparison. You define a baseline window that represents the distribution at launch or during a known-good period. You take a live window of recent traffic. You compare the two with a statistical test such as PSI or the Kolmogorov-Smirnov test. And you alert when the divergence between them crosses a chosen threshold.

The diagram's purpose is to show that drift detection is fundamentally about comparing distributions over time, not inspecting individual requests. A single request can't be 'drifted' — drift is a property of how the population of inputs or outputs has shifted relative to a reference. Framing it as baseline-versus-live with a test and a threshold makes it implementable, which the PSI code slide then delivers.

This slide grounds drift detection in practice by specifying what you actually measure and how you decide it matters. You pick features that capture meaningful aspects of behavior — input length, clusters of input embeddings, the distribution of judge scores, the refusal rate — and compute each feature's distribution over a baseline window. Then you compare recent traffic against that baseline using a statistical test like PSI or KS.

When the divergence crosses a threshold, you alert: the inputs or the outputs have shifted, even though no one changed the code. That last clause is the whole value. Drift detection is your early-warning system for the silent, gradual degradation that post two described with the nine-week timeline. It catches the kind of decline that has no error and no deploy to blame, surfacing it as a measurable, alertable signal.

This code slide implements the Population Stability Index, the workhorse metric for drift on a single feature. It bins the baseline values, computes the fraction of baseline and live observations falling in each bin, clips the fractions away from zero to avoid taking the log of zero, and sums each bin's contribution of the difference in proportions times the log of their ratio.

The interpretive rule in the comment is what makes it usable: a PSI above roughly 0.2 indicates significant drift worth investigating, while smaller values are normal fluctuation. PSI is popular precisely because it reduces a whole distributional shift to one interpretable number you can chart over time and threshold for alerts. Running it nightly on features like input length or judge score is a lightweight, label-free way to catch drift, which is exactly what the code post's nightly job does.

This slide closes the mechanics with the highest-value signal of all: the users themselves. Explicit feedback — thumbs up or down — is the obvious form, but implicit signals are often richer: did the user edit your output, did they copy it, did they immediately retry, did they abandon the session. All of these carry information about whether the answer was actually good.

The key move is to pipe that feedback back into your traces. A thumbs-down should do double duty: flag that specific request for judge scoring so you understand what went wrong, and accumulate into a growing labeled dataset. Over time that dataset becomes two things you badly need — an evaluation suite drawn from real traffic and a source of fine-tuning data. This is how the monitoring loop feeds back into improving the model, closing the cycle the earlier diagram showed.

The closing mistake warns against the central trap of automated quality measurement: trusting the judge blindly. An LLM judge is itself a model, so it inherits model failure modes. It tends to favor longer answers, it can be swayed by a confident tone regardless of correctness, and it may rate outputs from its own model family more generously.

If you never check the judge against human labels, you are monitoring with an unvalidated instrument — your quality numbers might be precise and completely wrong. The fix is calibration: hand-rate a few hundred examples, measure the judge's agreement with those human ratings, and re-validate whenever you change the judge model or its prompt. The code post turns this into an actual agreement metric. A judge you haven't calibrated is a guess wearing the costume of a measurement.

The CTA hands off to post four, the code-heavy build, which turns these mechanics into working infrastructure: a tracing decorator, cost logging, a sampled judge scorer, a nightly drift job, the dashboard SQL, and an alert rule. The teaser promises concrete, copyable patterns that assemble the engine room into a running system.