Reinforcement Learning

Reinforcement learning is one of the three great branches of machine learning, alongside supervised and unsupervised learning, and it is the one that maps most directly onto how animals and people actually learn. The dog-and-treat analogy is not a simplification for beginners; it is genuinely the core idea. An agent acts, the world responds, a reward or penalty follows, and over many repetitions the agent's behavior shifts toward whatever earns the most reward.

This first post deliberately stays at the level of vocabulary and intuition. The temptation in RL is to jump straight to equations, but the equations only make sense once you can name the pieces. Get agent, environment, state, action, reward, policy, and return clear in your head and the rest of the series has somewhere to attach.

The defining feature of reinforcement learning is the absence of an answer key. In supervised learning, every training example comes with the correct output stapled to it. In RL, the agent is never told the right action. It is only told, after the fact, how much reward an action produced — and even that signal can be delayed, noisy, or sparse.

That single difference reshapes everything. The agent has to generate its own experience by acting, has to figure out which of its many actions deserve credit for an eventual reward, and has to keep exploring to discover better options. This is why RL is both more powerful and more fragile than supervised learning: it can solve problems with no labels at all, but it has far less to lean on.

These five words are the entire grammar of RL, so it is worth slowing down on each. The agent is the decision-maker you are training. The environment is everything outside the agent — the game board, the physics simulator, the road, the user. The state is a snapshot of the relevant situation at one moment. The action is one of the choices available to the agent. The reward is a single number the environment hands back saying how good that step was.

Notice that the reward is just a scalar. It does not say what the right action was, only how the chosen one scored. Designing that reward signal well is one of the hardest and most consequential parts of any RL project, a theme we return to in post 5.

The cycle diagram captures the heartbeat of every RL system. The agent observes the current state, its policy chooses an action, the environment transitions to a new state and emits a reward, and the loop repeats. This perception-action-feedback loop runs thousands or millions of times during training.

What makes it learning rather than just looping is that each turn through the cycle leaves the agent slightly changed. The reward and the new state are used to update the agent's internal estimates or policy, so the next time it sees a similar state it acts a little more wisely. Over enough cycles, random flailing turns into competent behavior.

A policy is the answer to the question 'what do I do now?' Formally it is a function from states to actions, sometimes deterministic (one action per state) and sometimes stochastic (a probability over actions). When people say an RL agent has 'learned', they almost always mean its policy has improved.

There are two broad ways to arrive at a good policy. Value-based methods first learn how good each state or action is, then act greedily with respect to those values. Policy-based methods adjust the policy directly. Either way, the policy is the artifact you ultimately deploy — it is the trained behavior, the thing that drives the robot or plays the game.

Delayed reward is what separates RL from a simple bandit problem and what makes it genuinely hard. When you choose an action, its true consequences may not show up for many steps. A chess move that looks like a blunder can be the setup for a checkmate twenty moves later. The agent has to learn to assign credit backward through that chain — the credit assignment problem.

This is also why a naive 'just maximize the next reward' agent fails badly at most interesting tasks. The agent must care about the long-run consequences of its actions, which is exactly what value functions and the Bellman equation, introduced in post 3, are designed to handle.

This snippet is the skeleton every RL program shares, no matter how sophisticated. You reset the environment to get a starting state, then loop: the policy picks an action, the environment steps forward and returns a new state, reward, and a 'done' flag, and the agent uses that feedback to learn. When done is true, the episode ends and you start over.

The APIs you'll meet later — Gymnasium in particular — follow this exact shape. env.reset() and env.step(action) are the two calls you will use constantly. Recognizing this loop now means the runnable code in post 4 will feel familiar rather than foreign.

Comparing RL to supervised learning sharpens what is distinctive about it. Supervised learning consumes a fixed, pre-labeled dataset and learns a direct input-to-output mapping; each prediction is independent of the others. RL has no labels, generates its own data by acting, and makes sequential decisions where each action changes the situation the next action faces.

The practical upshot: if you have labeled examples and the task is one-shot prediction, supervised learning is simpler and more reliable. RL earns its complexity only when the problem is genuinely sequential and no answer key exists. Knowing this boundary keeps you from reaching for RL when a classifier would do.

The return is the quantity an RL agent actually optimizes. It is not the immediate reward but the sum of all future rewards from now to the end of the episode. Because a reward stream could in principle be infinite, and because near-term reward is usually more valuable than distant reward, we discount: each step into the future is multiplied by gamma, a number slightly below one.

Discounting has two jobs. Mathematically it keeps the return finite and the algorithms stable. Behaviorally it sets the agent's time horizon: a gamma near 1 makes a far-sighted, patient agent, while a low gamma makes a short-sighted one that grabs quick rewards. Choosing gamma is therefore choosing how much the agent cares about the future.

This flow diagram shows the data path that ties the five core concepts together. The agent, holding its policy, emits an action. That action enters the environment, which responds with a new state and a reward. Those two signals flow back to the agent, which uses them to update its policy. Then the cycle starts again.

Seeing it as a closed loop rather than a one-way pipeline is important. In supervised learning, data flows once from dataset to model. In RL, the agent's own actions shape the data it sees next, which creates feedback effects — both the virtuous kind that drives learning and the pathological kind, like reward hacking, that we cover in post 5.

These examples are deliberately spread across very different domains to show RL's reach. AlphaGo and its successors conquered board games long thought to require human intuition. Robotics labs use RL to teach legged robots to walk and arms to grasp. Self-driving stacks use it for sequential decisions like lane changes. Recommendation systems frame engagement as a sequential reward problem.

The most consequential recent example is RLHF — reinforcement learning from human feedback — which is used to align large language models with human preferences. If you have chatted with a modern AI assistant, you have used a product shaped by RL, which is a good reason to understand how it works.

This closing slide points ahead to the next post in the day, which makes the case for why RL is worth the trouble despite being harder than supervised learning. Saving and following keeps the five-post arc together: concept, then why it matters, then how it works, then code, then the mistakes that bite.

The teaser frames post 2 as the 'so what' question, the problems RL cracks that no labeled dataset ever could. Having the definitions from this post locked in is exactly what makes that motivation land, because you will be able to see why goals-and-consequences learning reaches places supervision cannot.