Serving Models with FastAPI

This post traces a request through a FastAPI serving app so the framework stops feeling like magic. The payoff is that once you see the four moving parts — the ASGI server, the schema, the loaded model, and the handler — and how they connect, you can reason about where latency comes from, why a handler hangs, and how to scale it. The cover frames the app as exactly those four parts wired together, and the rest of the post follows a request through them.

The stack diagram lays out the serving layers from top to bottom: the client sends an HTTP request, Uvicorn (the ASGI server) owns the network and the event loop, the FastAPI app handles routing and validation, your handler holds the predict logic, and the model sits at the bottom as an in-memory object. Each layer talks only to its neighbors.

Seeing the stack explicitly is the organizing idea for the whole post — each following slide zooms into one layer's job. The deeper point is the separation of concerns: Uvicorn knows nothing about your model, your model knows nothing about HTTP, and FastAPI is the translator in the middle. That layering is exactly what lets you swap the model, change the server, or restructure a handler without the other layers caring.

The first mechanic is that Uvicorn, not FastAPI, is the server. Uvicorn owns the socket and the event loop: it accepts incoming TCP connections, parses raw HTTP bytes into structured request objects, and hands them to your FastAPI application. FastAPI itself is just the application Uvicorn runs — which is precisely why you launch with 'uvicorn main:app', naming the module and the app object.

This separation matters more than it first appears. It's the ASGI contract: any ASGI server (Uvicorn, Hypercorn, Daphne) can run any ASGI app (FastAPI, Starlette), so the server and the framework are independent choices. Understanding that the server is a separate layer is what makes deployment concepts later — workers, Gunicorn managing Uvicorn processes — make sense rather than seem like arbitrary incantations.

The second mechanic is loading the model once, on startup, before any request arrives. FastAPI's lifespan handler runs exactly once when the process boots: it loads the model into memory, stores it where handlers can reach it, and yields control to start serving. The same model object then serves every request for the life of the process.

The alternative — loading inside the request handler — re-reads the file and reconstructs the model on every call, adding hundreds of milliseconds of pure overhead to each prediction and crushing throughput. This is the single most common serving performance mistake, which is why it gets its own slide here and reappears as mistake number one in the final post. The principle generalizes to any expensive resource: open it once at startup, reuse it across requests.

The code shows the lifespan pattern concretely. An async context manager decorated with @asynccontextmanager loads the model with joblib into a module-level dict before the yield, and clears it after — the part after yield runs on shutdown for cleanup. The FastAPI app is constructed with lifespan=lifespan so the framework runs this once at startup and once at teardown.

Two details carry the lesson. First, the model is stored in a dict (ml["model"]) rather than a bare global, which is a clean, testable way to share startup state with handlers. Second, everything before yield is startup and everything after is shutdown — the same construct manages both ends of the process lifecycle. This is the modern, recommended replacement for the older @app.on_event("startup") decorators, and it's the canonical place to put any load-once initialization.

The third mechanic is routing and validation. When a request arrives, FastAPI matches its URL and HTTP method to the right handler, then inspects that handler's type hints. Because the handler declares a Pydantic model as its parameter, FastAPI parses the request body into that model and validates it. If validation fails, FastAPI returns a 422 with field-level error details and your handler code never executes.

This is the gate that protects inference. The handler is written as if its input is already clean — and it genuinely is, because nothing malformed gets past the validation step. That guarantee is what lets the inference code stay simple: no defensive type-checking, no manual field validation, just the assumption that the typed object it received is well-formed, enforced by the framework upstream.

The code shows the handler doing inference on validated input. A Features Pydantic model declares a list of floats; the handler reshapes that list into the 2D array scikit-learn expects, calls the model's predict, and returns the result wrapped in a JSON-serializable dict. By the time this code runs, f.values is guaranteed to be a list of floats — validation already happened.

The load-bearing details are the reshape and the float() cast. Models expect a specific input shape (here, one row of features), so reshaping the flat list into (1, -1) is what makes predict accept it. Casting the prediction to a native float ensures it serializes cleanly to JSON — a NumPy scalar would either fail or leak an unstable type. These two small conversions are where a lot of serving bugs actually live, which is why they're worth seeing explicitly.

The fourth mechanic is the sync-versus-async handler decision, the subtlest and most consequential choice in the whole app. Define a handler with plain 'def' and FastAPI runs it in a threadpool, so CPU-bound inference doesn't block the shared event loop. Define it with 'async def' only when the body actually awaits asynchronous I/O. Getting this backwards — wrapping blocking CPU work in async def — silently serializes the entire server, because the blocked coroutine holds the event loop and no other request can progress.

This is counterintuitive: 'async' sounds faster, so people reach for it by default. But async only helps when there's something to await; pure CPU work in an async handler is strictly worse than the sync version, because FastAPI can't move it off the loop. The rule of thumb — def for CPU, async def for awaited I/O — prevents one of the most damaging and hardest-to-diagnose serving performance bugs.

The decision tree turns the sync/async rule into a flowchart. Does the handler await I/O like a database, HTTP call, or queue? If no, use plain def — it's pure CPU inference and the threadpool handles it. If yes, is the I/O library async-native? If yes, use async def with await; if the library is blocking, fall back to plain def so the blocking call runs in the threadpool rather than stalling the loop.

The subtlety the tree captures is that 'I have I/O' isn't enough to justify async def — the I/O library also has to support async. Using async def while calling a blocking, synchronous database driver gives you the worst of both worlds: the syntax of async with the blocking behavior of sync, jamming the event loop. Following the tree mechanically avoids that trap, which is exactly the kind of mistake that only shows up under concurrent load.

The fifth mechanic is the typed response. The handler returns a dict or, better, a Pydantic response model, and FastAPI serializes it to JSON. Declaring a response_model gives you three things at once: a documented output shape in the auto-generated docs, validation that your handler actually returns that shape, and automatic stripping of any extra fields you didn't declare.

The value is a predictable contract for callers. Without a declared response shape, an output can silently change — a refactor adds a field, a NumPy type leaks through — and downstream consumers break. With response_model, the output is pinned: the same fields, the same types, every time, enforced by the framework. It's the response-side mirror of input validation, closing the loop so both ends of the request are typed and guaranteed.

The flow diagram assembles all five mechanics into the full request path: Uvicorn accepts and parses the connection, FastAPI routes it to the matching handler, Pydantic validates the body, the handler runs model.predict, and the result is serialized to JSON on the way out. It's the same lifecycle from the concept post, now annotated with the components that own each step.

Seeing the complete flow is what makes debugging tractable. A 422 means the failure is at the Pydantic step; a hang under load points at a sync/async mistake in the handler; a serialization error lives at the final step. Because each stage has one clearly-owned job, an incident becomes a matter of identifying which stage broke rather than searching the whole app. That diagnostic map is the real payoff of understanding the mechanics.

Tracing a request through the server, model loading, validation, the handler, and the typed response gives you a working model of the app rather than a set of snippets to copy. Every stage has one job, and the connections between them — especially load-once-at-startup and the sync/async decision — are where performance and correctness are actually won or lost.

The next post turns this map into a complete, runnable application: it trains and saves a real model, defines typed request and response schemas, loads the model once with lifespan, adds a health check, and tests the whole thing with curl — so the mechanics stop being a diagram and become something you can launch.