Monitoring LLMs in Production

This cover sets the tone for the failure manual: broken LLM monitoring almost never means the tooling failed — it means the team measured the wrong things, drowned in noise, or logged data they shouldn't have. That reframing is encouraging, because every fix that follows is within reach and mostly a matter of judgment rather than technology.

Post five completes the series by cataloging six mistakes that most often make monitoring fail in practice: watching infrastructure instead of output, alert fatigue from bad thresholds, trusting an uncalibrated judge, ignoring cost until the invoice, logging PII, and collecting metrics nobody acts on. Each is common, each is fixable, and each quietly costs trust, money, or a 2 a.m. page that means nothing.

The first mistake is the most common: watching infrastructure instead of output. A team reuses its existing uptime, CPU, and error-rate dashboards, sees them all green, and concludes the LLM is healthy. Those metrics confirm the server is alive; they say nothing about whether the answers are correct, grounded, or safe.

The failure this misses is exactly the silent one from post one — a model can hallucinate every response while every infra metric stays green, because hallucination produces a perfectly healthy 200. The fix is to monitor the content: quality scores, groundedness checks, safety signals. Infra monitoring isn't wrong, it's just insufficient — it watches the pipe and ignores what flows through it. This is the mistake that makes all the others moot, because if you're only watching infra you won't even know the rest are happening.

The comparison puts infra-only and output-aware monitoring side by side so the gap is unmissable. The infra-only column tracks uptime, error rate, and p95 latency, looks green even while answers are wrong, and misses drift entirely. The output-aware column adds judge scores and groundedness, cost per request, catches silent failures, and detects drift over time.

The value of the contrast is as a quick self-audit. A team can look at its own dashboard and immediately see which column it lives in. Crucially, output-aware doesn't mean abandoning infra metrics — you keep them and add the content layer on top. The left column isn't wrong, it's incomplete, and the right column is what completes it into monitoring that actually catches LLM-specific failures.

The second mistake is alert fatigue, which destroys monitoring from the opposite direction: too much signal instead of too little. Teams set aggressive thresholds, get paged by noise — a single bad response, a momentary latency blip — and within a week the alert channel is muted. Then a real incident arrives and slips through the silence everyone has learned to ignore.

The fix is discipline about what deserves a page. Alert only on actionable, sustained signals: a quality drop held for fifteen minutes, not one low score; spend that's trending up, not a single expensive request. Page for what genuinely needs a human right now, and route everything else to a dashboard for review. The code post's 'for: 15m' alert clause is exactly this principle in practice — the difference between an alert people trust and one they mute.

The third mistake carries the judge warning from the mechanics post into the failure manual, because it's that important. An LLM judge is itself a model with measurable biases: it tends to prefer longer answers, it rewards a confident tone independent of correctness, and it may rate its own model family more favorably. If you wire it up and never calibrate it, the quality numbers on your dashboard are fiction dressed as data.

The fix is calibration as an ongoing practice, not a one-time step. Hand-rate a few hundred examples to get human labels, measure how well the judge agrees with them, and re-validate whenever you change the judge model or its rubric. An uncalibrated judge is worse than no judge, because it gives you false confidence — you think you're measuring quality when you're measuring the judge's quirks.

This code slide makes judge calibration concrete and quantitative. It takes a sample of human ratings and the judge's scores for the same items and computes Cohen's kappa with quadratic weighting — a standard agreement statistic that accounts for the ordinal distance between scores, so being off by one counts less than being off by three.

The interpretive rule in the comment is what makes it actionable: a weighted kappa below roughly 0.4 means the judge is unreliable and the rubric needs fixing before you trust its numbers. Quadratic weighting is the right choice here precisely because these are graded scores, not categories — a judge that says 4 when a human says 5 is nearly right, and the metric should reflect that. Tracking this agreement number over time, and especially after any change to the judge, is how you keep your automated quality measurement honest.

The fourth mistake is ignoring cost until the invoice — treating spend as a finance concern rather than an engineering metric. This works fine right up until a runaway agent loop or an unbounded context window multiplies the bill over a weekend, and you discover it weeks later when the invoice lands.

The fix mirrors the argument from post two: track cost per request and per feature in real time, set per-request budget guards that alert when a single call is abnormally expensive, and treat a spend spike with the same urgency as an error spike. The principle is that the invoice should merely confirm what your dashboard already told you days earlier. Cost is variable, can spike without any error, and is fully visible from the token counts you're already logging — there's no excuse for learning about it from accounting.

The fifth mistake is logging PII into your logs, which turns the monitoring system itself into a compliance liability. Capturing raw prompts and responses for replay is genuinely useful, but those prompts contain names, emails, health details, and account numbers, and dropping them into logs the whole organization can query is a breach waiting to be reported.

The fix is to treat trace data as sensitive by default: redact or hash PII at the edge before it ever reaches storage, set short retention windows so sensitive data doesn't pile up, and restrict who can read traces. The tension with debugging is real — you want enough context to reconstruct a failure without storing identifiers you'd have to disclose. The next slide shows a concrete redaction function. A debugging convenience must never become the incident you have to report to regulators and users.

This code slide implements PII redaction at the logging boundary. It defines regular expressions for common sensitive patterns — emails, phone numbers, and social security numbers — and a redact function that replaces each match with a typed placeholder like [email], then applies it to the prompt before it's logged.

The design is intentionally simple so it's easy to adopt and extend: add a pattern for whatever identifiers your domain involves. Replacing with a typed tag rather than just deleting is a nice touch — it preserves the structure of the prompt for debugging while removing the sensitive value, so a trace remains useful without being a liability. Regex redaction isn't perfect and won't catch everything, but applying it at the edge, before data reaches storage, is the high-leverage first line of defense that prevents the most common and most embarrassing leaks.

The sixth mistake is the most subtle because it can coexist with a beautiful setup: collecting metrics nobody acts on. A dashboard with forty charts, no owner, and no alerts is theater — it looks like monitoring and produces nothing. If a quality drop has no alert, no runbook, and no responsible person, then monitoring is decoration.

The fix is to tie every metric to an action. Define a small set of metrics that matter, and for each one specify the trigger and the response: when this number crosses this line, this person does this thing. This closes the loop the mechanics post's cycle diagram ended on — the 'act' step. Monitoring only pays off when it drives a fix, and the discipline of fewer metrics, each with an owner and a runbook, beats a wall of charts that no one watches and no one acts on.

The tips slide condenses the entire failure manual into a six-line checklist: monitor output quality not just infra, alert on sustained and actionable signals, calibrate the judge against humans, track cost as an engineering metric, redact PII before storage, and tie every metric to an action.

These six lines are the practical residue of the whole series. A reader who internalizes them will avoid the great majority of LLM monitoring failures and will build a system that catches silent regressions, controls cost, respects privacy, and actually drives fixes — rather than one that looks impressive and protects nothing.

The diagram contrasts noisy monitoring with useful monitoring as a final synthesis. Noisy monitoring pages on every blip, sprawls into forty unowned charts, runs an uncalibrated judge, and has no runbook. Useful monitoring has a few actionable alerts, metrics tied to fixes, a validated judge, and a clear owner with a runbook.

The point of ending on this contrast is that the difference between monitoring that works and monitoring that wastes everyone's time is rarely the tooling — it's these judgment calls. Every item in the right column is a decision, not a product. That's an empowering note to close a series on: the path from theater to genuine reliability is choices you can make today with the tools you already have.

The CTA closes both this post and the run-up to Day 100, framing the five posts as a complete, in-depth treatment of monitoring LLMs in production. It hands off to the series finale, which turns everything built across the hundred days into a tangible AI portfolio — the capstone that makes the learning visible to employers and collaborators.