Reinforcement Learning

This post answers the 'so what' question. Post 1 told you what RL is; this one argues why it earns a place in your toolkit despite being harder to use than supervised learning. The short version: RL is the only mainstream approach that learns goal-directed behavior over time without needing someone to label the correct action in every situation.

That capability unlocks a whole class of problems — games, robotics, control, alignment — where the right answer is either unknown, context-dependent, or impossible to enumerate. Understanding why RL matters is also how you develop the judgment to know when not to use it, which is just as valuable.

The deepest reason RL exists is that for many important problems, labels simply run out. Supervised learning assumes someone can provide the correct output for each input. But nobody can hand-label the optimal action for every one of the roughly 10^170 possible Go positions, or for every traffic situation a car might face. The space is too large and the 'right' answer too situational.

RL replaces the demand for labels with a far weaker requirement: a reward signal and a way to act. You don't tell the agent what to do; you tell it what you want (high reward) and let it discover how. This shift — from supervision to goals — is what lets RL tackle problems supervised learning structurally cannot.

Where most ML models output a single prediction about a single input, RL learns an entire policy: a coherent strategy for behaving across many connected decisions. This is a qualitatively different kind of output. A policy can plan, defer gratification, set up future opportunities, and adapt as conditions change mid-episode.

That is exactly the shape of control and game problems. A trading strategy, a robot gait, a game plan — none of these are single predictions; they are policies unfolding over time. RL is the natural language for any task where the question is 'how should I behave over a sequence of steps?' rather than 'what is the label for this one input?'

Self-play is RL's most dramatic demonstration of power. Because the agent generates its own training data by acting, it is not capped by the quality of human examples. AlphaGo trained partly on human games, but AlphaZero learned Go, chess, and shogi entirely from self-play, starting from random moves and surpassing all prior programs and players.

The famous move 37 in AlphaGo's match against Lee Sedol is the canonical illustration: a move human experts initially judged a mistake, which turned out to be brilliant. RL is not constrained to imitate; it can explore regions of strategy space no human ever visited, which is why it can produce genuinely superhuman and sometimes alien-looking play.

This comparison frames the practical decision an engineer faces. A hand-coded controller encodes a human's understanding directly: every rule is written out. It is interpretable and predictable, but brittle the moment reality steps outside the engineer's anticipated cases, and it can never exceed the insight of the person who wrote it.

An RL agent instead learns from reward. It adapts to states the designer never explicitly considered and improves as it gathers more experience, and it can discover strategies the designer never imagined. The cost is unpredictability and the difficulty of training. The choice between them is really a choice between control and capability.

These real deployments ground RL's value beyond games. Robotics is the obvious one — locomotion and manipulation are sequential control problems that RL handles well. Less obvious but economically large: Google used RL-style optimization to cut datacenter cooling energy substantially, and chip-placement work used RL to help lay out parts of hardware.

The common thread across the mindmap is that each domain involves a sequence of decisions whose effects compound over time toward a long-term objective. Wherever that structure appears — and it appears in logistics, energy, finance, and systems control — RL has a credible claim to be the right tool, provided you can simulate or sample the environment affordably.

RLHF is probably the most widely felt application of RL today, even though most people have never heard the term. After a large language model is pretrained on text, it is fine-tuned using human feedback: people rank model responses, a reward model learns to predict those preferences, and RL optimizes the language model to produce responses that score well.

This is why modern assistants feel helpful, follow instructions, and mostly avoid harmful output. The reward signal here is human preference rather than a game score, but the machinery is the same reward-driven policy improvement covered throughout this series. RL did not just win at Go; it shaped the AI products millions use daily.

This snippet drives home the single most important practical fact about RL: what you reward is what you get. The agent has no access to your intentions, only to the reward number you defined. Change the reward and you change the goal, often dramatically, even with the same agent and environment.

In the example, adding a fuel penalty turns a 'reach the goal as fast as possible' agent into one that balances speed against efficiency. This expressiveness is a strength — you can specify rich objectives — but it is also a trap, because a poorly specified reward leads to the reward-hacking failures detailed in post 5. Reward design is the real craft of applied RL.

Honesty about RL's weaknesses is part of understanding why it matters, because it tells you when it does not. RL is notoriously sample-inefficient, often needing millions of interactions; it is unstable, sensitive to seeds and hyperparameters; and it is only as good as its reward function. None of these are minor footnotes — they are why RL has fewer production deployments than supervised learning despite its higher ceiling.

The rule of thumb: if you already have labeled data, prefer supervised learning. If the reward is hard to specify, or if trying a bad action is catastrophic and unsimulable, be very cautious. RL is a specialized power tool, not a default.

These criteria turn the post's argument into a decision checklist. RL is worth its complexity when decisions are sequential, feedback is delayed rather than immediate, no labeled answer key exists, you can simulate or sample the environment cheaply, and the goal is genuinely to achieve an outcome rather than to imitate humans.

If only some of these hold, reconsider. Cheap simulation in particular is often the deciding factor in practice: RL's hunger for samples is far more tolerable in a fast simulator than on slow, costly, or dangerous real-world hardware. Use this list to sanity-check whether a problem is actually an RL problem before committing to one.

Treat these five conditions as a gate rather than a wish list. The strongest signal that RL is appropriate is the combination of sequential decisions and delayed feedback with no available answer key, since that is precisely the regime supervised learning cannot serve. Cheap simulation is the practical multiplier: it converts RL's appetite for samples from a dealbreaker into a manageable compute cost.

When the goal is to exceed human performance rather than imitate it, RL becomes not just viable but uniquely capable, as the self-play results show. If most of these boxes go unchecked, a simpler supervised or rule-based approach will usually get you there faster and with far less pain.

This closing slide hands off to post 3, which opens the engine room. Saving and following keeps the arc intact: you now know what RL is and why it matters, and next comes the machinery that actually makes an agent learn.

The teaser names the three pillars of that machinery: value functions, the Bellman equation, and the explore-exploit tradeoff. Those are the concepts that turn the motivation in this post into a working algorithm, so they are the natural next step before any code.