Reinforcement Learning

This is the engine-room post. Having established what RL is and why it matters, we now open the hood on the machinery shared by nearly every RL algorithm. The good news is that the core ideas are few: value functions, the Bellman equation, the explore-exploit tradeoff, and an update rule that ties them together. Master these and the proliferation of algorithm names — Q-learning, SARSA, DQN, PPO, A3C — stops being intimidating.

The through-line is simple: estimate how good states and actions are, update those estimates from experience using the Bellman relationship, and act greedily on them while exploring enough to keep improving. Everything else is engineering on top of that foundation.

The value function is RL's answer to the delayed-reward problem from post 1. Instead of reacting only to immediate reward, the agent learns to estimate the expected total future reward — the return — obtainable from each state, assuming good play from there onward. This converts a far-off, hard-to-attribute payoff into a concrete number attached to the present moment.

There are two flavors. The state-value V(s) estimates the value of being in a state. The action-value Q(s,a) estimates the value of taking a specific action in a state. Q is especially useful because once you have it, choosing the best action is trivial: pick the action with the highest Q. Learning Q accurately is the whole game for value-based methods.

The Bellman equation is the mathematical heart of RL. It states a self-consistency condition: the value of a state equals the immediate reward you get plus the discounted value of wherever you land next. Written for action-values, Q(s,a) = reward + gamma * max over next actions of Q(next state, a').

What makes this powerful is the recursion. Because each state's value is defined in terms of its successor's value, reward earned late in an episode can propagate backward, step by step, raising the estimated value of the earlier states and actions that led to it. This backward flow is precisely how the agent solves credit assignment — how it learns that an early move was good because of a reward that came much later.

This flow diagram visualizes the backup operation that the Bellman equation performs. Imagine a chain of states A to B to C, where reaching C yields a reward. Initially the agent has no idea A or B were valuable. But once it experiences the reward at C, the Bellman update raises C's value, then on later passes B's value (because B leads to C), then A's (because A leads to B).

This is why RL often needs many episodes: information propagates one step backward per update, so a reward at the end of a long sequence takes many passes to reach the beginning. It is also why discounting and good exploration matter — they control how strongly and how far that value signal flows back through the chain.

The explore-exploit dilemma is unique to learning-by-acting and has no analog in supervised learning. At every step the agent faces a choice: exploit its current knowledge by taking the action it currently believes is best, or explore by trying something uncertain in the hope of discovering something better. Pure exploitation can lock in a mediocre habit; pure exploration never benefits from what it learns.

The simplest practical solution is epsilon-greedy: with probability epsilon, act randomly; otherwise act greedily. You typically start with high epsilon (explore a lot early when you know nothing) and decay it over time (exploit more as your estimates sharpen). Tuning this schedule is one of the most common things that makes or breaks an RL run.

This is the canonical Q-learning update, and it rewards close reading. Q[s][a] is the current estimate of how good action a is in state s. best_next is the value of the best action available in the next state — the agent's optimism about the future. target combines the actual reward just received with that discounted future value, giving a better estimate than the old one.

The final line moves the old estimate a fraction alpha of the way toward the target. The quantity (target - old) is the temporal-difference error — the surprise. If reality matched expectation, the error is zero and nothing changes; if it was better or worse than expected, Q shifts accordingly. This single line, applied millions of times, is enough to learn optimal behavior in many environments.

The cycle diagram shows how the pieces operate together each step. The agent acts using epsilon-greedy (exploit the current best, occasionally explore). It observes the resulting reward and next state. It performs a Bellman backup to update its Q estimate. And as its estimates improve, its greedy choices become better, so the policy implicitly improves.

This is a tighter, per-step view of the agent-environment loop from post 1, now annotated with the specific learning mechanism. Notice there is no separate 'training phase' and 'deployment phase' as in supervised learning — in classic RL, acting and learning are interleaved on every single step.

Generalized policy iteration is the name for the virtuous cycle that makes RL converge. It has two interacting processes: policy evaluation (making the value estimates accurate for the current policy) and policy improvement (making the policy greedy with respect to the current values). Each improves the other.

As value estimates get more accurate, acting greedily on them yields better behavior. Better behavior generates more informative experience, which sharpens the value estimates further, which enables even better behavior. Starting from a random agent, this loop ratchets upward toward an optimal policy. Almost every RL algorithm is some instantiation of this dance between evaluating and improving.

The model-free versus model-based split is the major architectural fork in RL. Model-free methods, like Q-learning, DQN, and PPO, learn purely from experienced rewards without ever building an explicit model of how the environment works. They are simpler and more general but sample-hungry, often needing huge numbers of interactions.

Model-based methods, like AlphaZero and MuZero, learn (or are given) a model of the environment's dynamics and use it to plan by simulating possible futures. This makes them far more sample-efficient — you can think before acting — at the cost of greater complexity and the risk that an inaccurate model misleads the planner. Knowing which family an algorithm belongs to tells you a lot about its tradeoffs.

This pipeline summarizes the full learning process end to end. You initialize the Q-table or policy as a blank slate. You collect experience by acting in the environment. You update your value estimates using Bellman backups. And you repeat, gradually converging toward an optimal policy as estimates stabilize and the policy stops changing.

The runnable code in post 4 implements exactly this pipeline on a concrete environment, so it is worth holding this four-step shape in mind. Init, collect, update, repeat — that is the loop you will write, and recognizing it now makes the code read like a story rather than a wall of syntax.

These five points are the minimum you should carry forward. Value is your estimate of expected future reward, which converts delayed payoffs into present-moment guidance. The Bellman equation is the recursion that propagates reward backward through states. Discounting makes future reward worth progressively less and keeps the math finite.

The explore-exploit balance governs how the agent gathers information versus cashes it in, and getting it wrong is a top cause of failure. Finally, acting greedily with respect to learned values is what turns those numbers into a policy. Hold these five and you can read almost any RL algorithm description and place its pieces.

Use this list as your compression of the whole engine room. Value is the bridge from delayed reward to present decision; Bellman is the recursion that builds those values; discounting tunes the horizon; explore-exploit governs how experience is gathered; and greedy action on values is what becomes the deployed policy. Every algorithm name you meet later is a different way of computing or approximating these same five things.

If you can recite this list and explain each item in a sentence, you have the conceptual core needed to read research papers and library docs without getting lost. The deep RL methods just swap the Q-table for a neural network and add tricks for stability; the skeleton underneath is exactly these five ideas.

This closing slide points to post 4, where all of this theory becomes a program you can run. Saving and following keeps the build connected to the concepts you just learned, so the code reads as a realization of the loop rather than mysterious syntax.

The teaser promises a complete Q-learning agent solving a real environment. Watching a Q-table go from all zeros to a competent policy is the moment the value functions, Bellman updates, and epsilon-greedy exploration stop being abstractions and start being lines you typed.