OpenAI & Claude APIs

This post traces a single request end to end so the API stops feeling like magic. The payoff is debuggability: once you can name each stage — tokenize, request, sample, stream, stop — you can localize any failure instead of guessing. The cover frames the whole post as following one call from your JSON to a stream of tokens and back.

Tokenization is the first and most underappreciated stage. A tokenizer specific to the model family converts your text into integer token ids. Common English words may be a single token, while rare words, code identifiers, and non-Latin scripts split into several. This is precisely why cost depends on the content of your text, not its character count.

Different providers use different tokenizers, so the identical string can cost a different number of tokens on OpenAI versus Claude. When you are estimating cost or fitting content into a context window, you must count tokens with the provider's own tokenizer rather than assuming a fixed characters-per-token ratio.

The pipeline diagram lays out the internal stages a request passes through on the server: your messages JSON is tokenized into ids, the model's forward pass produces logits (raw scores) for every possible next token, a sampler selects one, and the chosen ids are detokenized back into text. The loop over the sampler stage is what produces a multi-token reply.

Seeing these stages explicitly demystifies behavior that otherwise looks arbitrary. Randomness in the output comes from the sampler stage; cost comes from the token counts at the ends; truncation comes from a stop condition in the loop. Each observable behavior maps to a specific stage.

The request body is where your control surface lives. You POST JSON with the model name, a messages array, and sampling parameters. The server validates your key, assembles the full prompt — including any system instructions and the entire conversation history you resent — and feeds it to the model.

Two common 400 errors originate here: malformed JSON, and for Claude specifically, a missing max_tokens. Because the API is stateless, forgetting to include prior turns is not an error but a behavior change — the model simply has no memory of a conversation you did not resend. Knowing the body is assembled server-side from exactly what you send clarifies both classes of problem.

Next-token sampling is the heart of generation. After the forward pass, the model holds a probability distribution over its entire vocabulary for the next token. The sampler picks one token from that distribution, shaped by temperature (which flattens or sharpens the probabilities) and top_p (which restricts sampling to the most likely tokens that sum to a probability mass). The picked token is appended and the process repeats.

This explains why output is generated one token at a time and why the same prompt can yield different answers at higher temperature. It also explains the tuning advice: lower temperature for deterministic, factual tasks; higher for creative ones. The model is not 'deciding' an answer up front — it is walking a probabilistic path token by token.

These are the knobs you actually set. temperature controls randomness, with lower values producing more deterministic output. top_p is the nucleus-sampling cutoff and is usually left at 1.0 unless you specifically want to truncate the tail. max_tokens caps the output length — and only the output, not the input. stop lets you halt generation when the model emits any of the given strings.

The most common confusion this slide preempts is that max_tokens limits the whole request; it does not. It bounds only what the model generates, which is also why setting it too low silently truncates replies, a problem revisited two slides later.

Streaming changes when you receive the text, not how much you pay. With stream=True the server emits tokens as they are generated using server-sent events, so your interface can render words progressively. The full response still costs the same number of tokens; you have simply chosen to consume them incrementally.

The user-experience benefit is large: time-to-first-token drops to a fraction of a second, making the system feel alive even on a long answer. The tradeoff is slightly more complex client code, since you handle a stream of events and must reassemble the final text yourself if you need it whole.

This is the concrete streaming pattern for Claude. The messages.stream context manager opens a streaming connection, and iterating stream.text_stream yields text fragments as they arrive, which you print immediately with flush=True so they appear in real time. The context manager cleanly closes the connection when the block exits.

The OpenAI equivalent uses stream=True on the create call and iterates chunks, reading delta content. The shapes differ but the principle is identical: consume tokens as they are produced. Showing the Claude form here keeps the day balanced across both providers and demonstrates the SDK's ergonomic streaming helper.

Stop reasons are the most practically useful piece of metadata the API returns, and the most ignored. Generation ends either because the model emitted an end-of-turn token — reported as finish_reason 'stop' on OpenAI or stop_reason 'end_turn' on Claude — or because it hit your max_tokens cap, reported as 'length' or 'max_tokens' respectively.

The operational habit this builds is to check the stop reason before trusting a reply. A response that looks truncated almost always carries a length/max_tokens stop reason, telling you exactly how to fix it: raise the limit and retry. Skipping this check is how truncated, broken output ends up parsed and shipped.

The cycle diagram captures the autoregressive generation loop in four beats: take the current tokens (prompt plus everything generated so far), run a forward pass to get logits, sample the next token using your temperature and top_p settings, then append it and check whether a stop condition is met — ending if so, otherwise repeating.

Visualizing it as a loop, rather than a single black-box step, explains both cost and latency. Every generated token is one full pass through this cycle, so output length drives both the bill (more output tokens) and the wall-clock time (more iterations). That single insight ties together the cost and latency themes from across the day.

The usage object is where the abstractions become numbers. Both APIs return token counts: input/prompt tokens for everything you sent, and output/completion tokens for what the model wrote. Many requests also report cached input tokens, which are billed at a reduced rate when prompt caching applies.

The discipline this slide promotes is to log usage on every call. Multiply the counts by your per-token prices and you have exact cost per request, which turns the cost discussions from earlier in the day into a dashboard rather than a guess. Usage logging is the cheapest observability you can add and the first thing you will wish you had when a bill surprises you.

Tracing one request end to end gives you a debugging map: tokenization explains cost and truncation; the request body explains 400s; sampling explains nondeterminism; streaming explains perceived speed; and stop reasons explain early endings. With that map, problems become locatable rather than mysterious.

The next post turns this understanding into running code — a single script that calls both providers, streams, uses tools, forces structured output, and handles errors the way production code should.