Serving Models with FastAPI

This is the build post, so the detail entries focus on why each piece of code is shaped the way it is, not just what it does. The throughline is that this is a complete serving app in miniature: train and save a model, declare typed schemas, load once on startup, add a health check, and test with curl. It's a paste-and-grow skeleton — swap the iris model for your own and the structure holds.

This step establishes setup the practical way: install the serving stack — FastAPI and Uvicorn for the API, scikit-learn for the model, joblib for persistence, NumPy for array handling — and import the model pieces. The imports are split between training (the dataset and classifier) and serialization (joblib), which previews the two-phase shape of the app: train once, serve many.

Keeping the dependency list explicit matters because serving apps are deployed, and a deployment needs a reproducible environment. In a real project these exact pins would live in a requirements file or Dockerfile. The small discipline of naming every dependency up front is what prevents the 'works on my laptop, missing on the server' class of failure when this skeleton moves from your machine to a container.

This step trains and saves the model — deliberately a separate script that runs once, not part of the API. It fits a random forest on the iris dataset and dumps it to iris.pkl with joblib. The comment makes the boundary explicit: after this runs, the model is an artifact on disk that the API can load, completely decoupled from training.

Separating training from serving is the load-bearing design choice. The API process should never train — training is slow, needs the full dataset, and happens on a different schedule than serving. By persisting the fitted model to a file, the serving app's only job becomes loading and predicting. This is the same artifact-handoff that real systems formalize with a model registry; here it's a .pkl file, but the principle — train elsewhere, serve from a saved artifact — is identical.

This step defines the typed contract on both ends. IrisInput declares features as a list of exactly four floats, using min_length and max_length so a request with the wrong number of features is rejected before inference. Prediction declares the output shape: a class id, a human-readable label, and a confidence score, all typed.

Declaring both an input and an output model is what makes the endpoint self-documenting and safe at both boundaries. The length constraint on the input is especially important for models — passing three or five features to a model expecting four is a classic source of a cryptic shape error deep in scikit-learn; here it's caught at the door with a clear message. The Prediction model means callers get a stable, documented response shape rather than whatever dict the handler happens to build.

This is the core of the app, fusing the load-once and predict-per-request patterns. The lifespan handler loads iris.pkl into a module-level dict once at startup and clears it on shutdown. The app is constructed with that lifespan. The /predict handler, declared with response_model=Prediction, reshapes the validated features, calls predict_proba, picks the most likely class, and returns a populated Prediction model.

Several deliberate choices stack up here. The model loads once and every request reuses ml["model"], avoiding per-request reload. The handler is plain def, so FastAPI runs the CPU-bound inference in a threadpool without blocking the event loop. predict_proba (not just predict) is used so the response can include a real confidence. And the explicit int() and float() casts convert NumPy scalars to native Python types that serialize cleanly — the exact conversions that prevent a whole category of serialization bugs.

This step adds a /health endpoint, which is unglamorous but essential for real deployment. It returns whether the model has finished loading — 'ready' once it's in memory, 'loading' otherwise — along with a boolean. The comment names the consumers: load balancers and Kubernetes liveness/readiness probes hit this before routing traffic to the instance.

The reason a health check matters is the startup window. A serving process is alive (the port is open) before the model finishes loading, and sending real prediction traffic during that gap produces errors. A readiness endpoint that reflects actual model state lets the orchestration layer hold traffic until the instance can genuinely serve. Without it, every deploy and every autoscale event risks a burst of failures during the load window — a problem that's invisible in local testing and obvious in production.

This step runs the server and calls it, closing the loop from code to a live API. Uvicorn launches the app bound to all interfaces on port 8000 — binding to 0.0.0.0 rather than localhost is what makes it reachable from outside the container or host. Then a curl POST sends a JSON body with the four iris features and the correct content-type header.

Showing the curl call matters because it demonstrates the language-agnostic contract in action: nothing about the caller knows it's talking to Python or scikit-learn — it's just an HTTP POST with JSON. The explicit Content-Type: application/json header is required for FastAPI to parse the body as JSON; omitting it is a common first-time stumble. This is the moment the saved .pkl becomes a service that any client, in any language, can use.

The trace diagram shows exactly what a round trip looks like: the POST with its JSON feature body goes in, the comment marks where validation and predict_proba happen, and a 200 OK comes back with the typed response — class_id 0, label setosa, confidence 0.99. It's the abstract request lifecycle from the concept post, now filled in with real values.

Seeing the concrete input and output side by side is what makes the contract tangible. The caller sends four numbers and gets back a structured, documented object — not a bare class index they'd have to interpret, but an id, a human label, and a confidence they can threshold on. That richer response is a direct payoff of using predict_proba and a typed response_model, and it's the difference between an endpoint that's merely functional and one that's genuinely usable by another team.

The flow diagram shows how the five pieces wire together at runtime: train.py produces iris.pkl, the lifespan handler loads it once, IrisInput validates each request body, predict() computes probabilities and a label, and the Prediction model shapes the typed JSON out. Each box has one job, which is what makes the skeleton extensible.

Seeing the composition clarifies where real-world additions slot in. Prediction logging wraps the predict box; authentication sits in front of the handler; a model registry replaces the bare .pkl in the load step; batching sits inside predict() when the model supports it. Because the stages are cleanly separated, each can be hardened independently — the simple app grows into a production service by thickening individual boxes rather than rewriting the flow.

The checklist condenses the post into the practices that turn this skeleton into something production-grade. Pin and log the model version at startup so you always know what's serving. Add both a shallow /health and a deeper /ready check that confirms the model can actually predict. Set request timeouts and a maximum body size so one bad request can't stall or balloon the server. Containerize with a Dockerfile and multiple Uvicorn workers. And log every prediction so you have the data to monitor for drift later.

None of these are difficult individually; the value is doing all of them, because each closes a gap that only appears under real traffic. The skeleton above is honest about being a skeleton — these are the specific reinforcements it needs before production, and each one maps directly to a failure mode the final post catalogs.

With a complete, runnable, validated serving app in hand, the natural next question is what still goes wrong even with a working endpoint — the operational mistakes that quietly wreck latency and reliability despite the model being fine. That's exactly where the final post goes.

The transition is intentional: you've now seen the right way to load, validate, respond, and health-check, so the mistakes post reads as a checklist of the specific ways teams deviate from these patterns — reloading the model per request, blocking the event loop, skipping validation — and pay for it in slow, fragile endpoints that fall over under load.