MLOps in 8 Slides

The closing post is a catalog of the operational mistakes that sink ML projects — chosen because they're common, not exotic. The framing is that most ML failures aren't bad models; they're bad operations: no versioning, no monitoring, a notebook in production. Learning these from a list is far cheaper than learning them during a 3am incident with a model nobody can rebuild. Each mistake is paired with a concrete fix so the post is a checklist, not just a warning.

The first mistake is versioning code but not data and models. It's the habit carried over from traditional software, where code alone determines behavior — but in ML, behavior comes from the data and weights too. Version only the code and six months later you can't answer 'what trained this?', which means you can't debug, audit, or safely roll back.

The fix is to treat data snapshots and model artifacts as first-class versioned things, exactly like code. A dataset gets a version or content hash; a trained model gets a registry version. Once all three are pinned together, any past result becomes reproducible on demand. This is the single most foundational MLOps practice, which is why it's mistake number one — almost every other failure gets worse without it.

The fix code contrasts the bad pattern — train(load_csv('data.csv')), where the data that produced the model is immediately forgotten — with the good one, where an MLflow run logs the data version and git SHA as params and registers the model. Now the run records all three artifacts: which code, which data, which resulting model.

The data_version string would typically come from a tool like DVC or a content hash of the dataset; the point is that it pins the exact snapshot, not just 'some CSV.' This is the runnable embodiment of the three-artifacts discipline from the first post, now framed as the fix to the most common reproducibility failure. The whole difference between an auditable model and an untraceable one is these few extra log lines.

The second mistake is deploy-and-forget: shipping a model and walking away as if the world will hold still. It won't. Without monitoring, drift erodes accuracy for months while the model keeps returning confident, increasingly wrong answers — the silent failure from the why-post, now framed as an operational omission rather than an abstract risk.

The fix is to make monitoring part of the deployment, not a later add-on: watch input drift, the prediction distribution, and live accuracy from day one. The key reframing is that a deploy isn't done when the model is serving — it's done when the model is serving and being watched. Treating monitoring as optional is what turns a normal model into one that's silently broken before anyone notices.

The compare diagram puts deploy-and-forget beside deploy-and-monitor. The forgotten model has no drift signal, rots silently, gets discovered through customer complaints, and forces a panicked retrain. The monitored model alerts on drift early, catches decay in days, surfaces problems on a dashboard, and retrains on a calm schedule. Same model, opposite operational experience.

The lesson is that monitoring changes when and how you find out — and 'when you find out' determines whether the fix is routine or a fire drill. Finding decay on a dashboard in days lets you retrain deliberately; finding it through complaints in months means the damage is already done and the response is reactive. Monitoring is what moves you from the right column's calm to avoiding the left column's panic.

The third mistake is training/serving skew: computing features one way during training (a clean pandas script) and another way during serving (a hurried API handler). Even tiny discrepancies in preprocessing — a different default, a rounding difference, a column order — produce predictions that don't match what the model learned, and the model appears broken for reasons that have nothing to do with the model.

The fix is to share the exact same transformation code between training and serving: one featurization function, imported by both paths, rather than two implementations that drift apart. This is one of the most insidious ML bugs because the model is fine and the symptoms point everywhere but the real cause. Eliminating the second implementation eliminates an entire class of 'works in training, fails in prod' mysteries.

The fix code shows the cure: a single featurize function, defined once in a shared module, that both the training path and the serving path call. Because the exact same code computes tenure_yrs and age_bucket in both places, there's no opportunity for the two paths to diverge. The comment makes the symmetry explicit — train calls featurize(train_df), serve calls featurize(request_df), same function.

The principle generalizes beyond this snippet: any transformation applied before the model — scaling, encoding, imputation, feature derivation — must come from shared code or a serialized pipeline object, never be reimplemented per environment. The moment training and serving have separate preprocessing implementations, they will eventually disagree, and the model will be blamed for a bug that lives in the glue.

The fourth mistake is treating a Jupyter notebook as a production pipeline. A notebook run top-to-bottom by hand has hidden cell-execution order, manual steps, and untracked in-memory state, which makes it impossible to reproduce reliably or automate. It works for the person who wrote it, on the day they wrote it, and almost nowhere else.

The fix is to refactor the proven notebook into versioned, parameterized scripts that run unattended in CI or an orchestrator. The notebook is a fine place to explore and discover; it's a terrible place to operate. Promoting from notebook to script is the step that turns a one-off result into a repeatable process — and skipping it is why so many promising experiments never become dependable systems.

The compare diagram contrasts the notebook in prod with a real pipeline. The notebook is run by hand, has hidden cell order, no tests and no CI, and is not reproducible. The pipeline runs on a trigger, has deterministic ordered steps, is tested and versioned, and is reproducible. The single most telling row is the last: reproducible, no versus yes.

The deeper point is that 'it ran for me' and 'it runs reliably for anyone, unattended' are completely different bars, and the gap between them is exactly the refactor from notebook to pipeline. Everything the rest of the day builds — tracking, gating, scheduling, monitoring — assumes deterministic, automatable steps, which a hand-run notebook fundamentally can't provide.

The fifth mistake is having no rollback plan: a new model ships, regresses in production, and there's no fast way back because the previous version wasn't kept as a deployable artifact. Now recovering means retraining from memory under pressure — the worst possible time to discover your old model is gone.

The fix is a model registry with staged versions, which keeps every version deployable so rollback is a one-line stage change rather than a rebuild. Better still, canary or shadow deploys let you expose a new model to a slice of traffic first, catching the regression before it reaches everyone. Rollback safety is the payoff of the registry discipline from the how-post — because every version is retained, going back is instant instead of catastrophic.

The fix code shows rollback as a single registry operation: re-promote the last known-good version to Production by transitioning its stage. There's no retraining, no file hunting, no rebuild — because the registry kept version 11 as a deployable artifact, restoring it is one call. The comment drives the point home: rollback is instant precisely because every version is kept.

This ties the whole day together. The serving layer loads whatever is Production, so this stage transition instantly redirects live traffic back to the good model without touching serving code. That clean separation — change the registry, not the deployment — is what makes rollback a calm, one-line action instead of an emergency, and it's why the registry is worth the discipline of keeping old versions around.

The final checklist is the deployable summary of the whole day. Code, data, and model are all versioned. Monitoring is wired up before launch, not after an incident. The same transform runs in training and serving. The pipeline runs unattended rather than by hand. The previous version is kept for instant rollback. And a metric gate blocks bad deploys before they ship.

Running down this list before deploying any ML system catches the great majority of operational incidents before they reach users. None of the items are difficult individually; the value is in doing all of them, because ML systems tend to fail on whichever check you skipped — and they fail silently, which is exactly why a deliberate pre-ship checklist matters more here than in normal software.

That closes Day 96. You now have the full arc: what MLOps is, why models decay and fail silently, how a pipeline moves a model to production and keeps it fresh, a runnable mini-pipeline with validation and gating, and the operational mistakes that sink ML projects.

The practical takeaway stands on its own — version code, data, and model; monitor from day one; share your transforms; automate the pipeline; and keep your last good version for instant rollback — and your ML systems will hold up under real traffic instead of quietly rotting after the demo.