Serving Models with FastAPI

The closing post is a catalog of the serving mistakes that quietly wreck FastAPI endpoints — chosen because they're common, not exotic. The framing is that most serving disasters aren't bugs in the model; they're operational slips around it: the model reloaded on every request, blocking work jammed into an async handler, no validation at all. Learning these from a list is far cheaper than discovering them during a load spike, so each mistake is paired with a concrete fix and the post functions as a pre-ship checklist.

The first mistake is loading the model inside the request handler. Every call to joblib.load() re-reads the file from disk and reconstructs the model object, adding hundreds of milliseconds of pure overhead to each request and destroying throughput under any real concurrency. It's the single most common reason a fast model becomes a slow endpoint.

The fix is the load-once pattern from the mechanics post: load the model a single time at startup in a lifespan handler, store it in memory, and have every request reuse that same object. The reframing is that model loading is startup work, not request work. Once you see the distinction, the mistake becomes obvious — you'd never reopen a database connection on every query, and the model is no different.

The fix code contrasts the two patterns directly. The bad version calls joblib.load() inside the handler, so the model is rebuilt on every request. The good version loads it once in the lifespan handler into a shared dict and the handler simply reuses ml["model"]. Same model, same prediction — but one pays the load cost once and the other pays it on every call.

The principle generalizes to any expensive per-process resource: tokenizers, feature transformers, database pools, downstream clients. Initialize them at startup and share them across requests. The comment in the good path points back to the lifespan handler precisely because that's the canonical home for all such load-once initialization. This one change is often the largest single latency win available in a naive serving app.

The second mistake is putting blocking, CPU-bound inference inside an 'async def' handler without awaiting anything. Because the coroutine never yields control, it holds the event loop for the entire duration of the inference, so the server processes requests strictly one at a time — no matter how many workers you believe you have. The async keyword gives a false sense of concurrency while delivering serialization.

The fix is to use plain 'def' for CPU-bound handlers, which makes FastAPI run them in a threadpool that doesn't block the loop, or to offload heavy work to a process pool. The counterintuitive rule — async is not automatically faster — is the whole lesson. async def only helps when the body awaits real I/O; for pure CPU inference it's strictly worse than sync, and the resulting bottleneck only appears under concurrent load, making it one of the hardest serving bugs to catch in testing.

The fix code shows the trap and the cure side by side. The bad version declares the handler async def and runs model.predict directly — CPU work with no await — which blocks the event loop for every other request. The good version is identical except it uses plain def, so FastAPI automatically runs it in a threadpool and the loop stays free to handle other requests.

The striking part is how small the difference looks: a single keyword. That's exactly why the mistake is so common and so damaging — it's invisible in a code review unless you know the rule, and it's invisible in single-request testing because the blocking only matters under concurrency. Internalizing 'def for CPU, async def for awaited I/O' is the cheapest possible insurance against a serving bottleneck that otherwise only reveals itself in production.

The third mistake is accepting untyped input — a raw dict or an unvalidated body — so malformed data flows straight into the model. A string where a float belongs, a missing field, or the wrong number of features produces either a cryptic 500 from deep inside the inference code or, worse, a silently wrong prediction computed on garbage. Either way the failure surfaces far from its cause.

The fix is a Pydantic model with field constraints, so bad input is rejected at the boundary with a clear 422 and field-level errors before inference runs. This is the validation payoff from the why-post, now framed as the cure to a specific failure. The model should only ever see data that already matches its expected shape and ranges, and the type declaration is what guarantees that without any hand-written validation code.

The compare diagram puts the two approaches side by side. With no validation, garbage reaches the model, you get cryptic 500s, sometimes silent wrong outputs, and debugging is hard because the error appears far from its source. With a Pydantic schema, bad input becomes a clean 422 with field-level errors, the model only ever sees valid data, and the schema doubles as documentation.

The lesson is that validation changes where and how clearly failures appear. Pushing the check to the boundary turns an investigation ('why did inference crash?') into an obvious, self-describing rejection ('field income must be greater than 0'). And because the same schema generates the API docs, you get correctness and documentation from one declaration — the recurring theme of why FastAPI is worth choosing for serving.

The fourth mistake is returning objects that don't serialize cleanly — a NumPy array, a tensor, or a raw model output. These either throw a serialization error or, when they do serialize, leak an unstable internal shape to callers who then depend on it. The output contract becomes accidental rather than designed.

The fix has two parts: convert predictions to native Python types (float, int, list) so JSON serialization is reliable, and declare a response_model so the output shape is typed, validated, and documented. The response_model also strips any extra fields, so the contract is exactly what you declared and nothing more. This is the response-side mirror of input validation — pinning the output shape so a refactor or a leaked NumPy type can't silently break downstream consumers.

The fix code shows the bad and good output patterns. The bad version returns a dict containing a raw NumPy array straight from model.predict, which doesn't serialize cleanly. The good version declares response_model=Prediction and returns a typed model built from explicit int() and float() conversions of the prediction and its probability.

The explicit casts are the load-bearing detail: NumPy scalars like np.int64 and np.float32 are not native Python types and don't reliably round-trip through JSON. Wrapping argmax in int() and the probability in float() produces clean, portable values. Combined with the response_model, the caller gets a guaranteed shape with guaranteed types on every response — turning the output from 'whatever the handler happened to build' into a stable, documented contract.

The fifth mistake is operating a single Uvicorn process with no concurrency, no limits, and no batching. One process can't use multiple CPU cores, so a multi-core machine sits mostly idle; with no request timeout, one slow request ties up a worker indefinitely; with no body-size cap, a huge payload can exhaust memory; and without micro-batching, a model that could score many rows at once handles them one at a time.

The fix is to run multiple workers (Uvicorn's --workers or Gunicorn managing Uvicorn worker processes) to use all cores, set request timeouts and a maximum body size to contain bad requests, and add micro-batching when the model supports it. These are the deployment-shaped reinforcements the build post's checklist pointed at — the difference between an endpoint that survives a traffic spike and one that quietly falls over under it.

The fix code shows the scaling and guarding knobs. Running Uvicorn with --workers 4 spawns multiple worker processes to use all CPU cores, and --timeout-keep-alive bounds idle connections. The Gunicorn variant manages Uvicorn workers with a hard request --timeout, which is the more battle-tested production launcher. The comment notes that a reverse proxy like nginx is where body-size limits typically live.

The key insight is that a single async process does not magically use all your cores — async handles concurrency within one process, but you still need multiple processes to use multiple CPUs for CPU-bound inference. Workers and async are complementary, not alternatives. Pairing multiple workers with timeouts, body limits, and a reverse proxy is what turns the development server into a deployment that holds up under real, adversarial traffic.

The final checklist is the deployable summary of the whole day. The model is loaded once at startup, not per request. Plain def is used for CPU inference, async only for awaited I/O. Pydantic validates every request body. Responses use native types and a response_model. Multiple workers, timeouts, and body limits are configured. And /health and /ready endpoints exist for orchestration probes.

Running down this list before deploying any FastAPI endpoint catches the great majority of serving incidents before they reach users. None of the items are difficult individually; the value is in doing all of them, because serving systems tend to fail on whichever check you skipped — and several of these failures only show up under concurrent production load, which is exactly why a deliberate pre-ship checklist matters more here than the happy-path code suggests.

That closes Day 98. You now have the full arc: what serving a model means and where FastAPI fits, why its validation and async concurrency pay off, how the server, schema, model loading, and handler connect internally, a complete runnable serving app, and the operational mistakes that wreck latency and reliability.

The practical takeaway stands on its own — load the model once, use def for CPU and async for I/O, validate input and type your responses, and run multiple workers with sane limits — and your model will serve as a fast, reliable API instead of a fragile script that collapses the first time real traffic arrives.