Reinforcement Learning

This is the hands-on post, and the goal is that you can copy each block in order and end up with a working, trained agent. We use Q-learning on FrozenLake, a classic tiny environment from Gymnasium (the maintained successor to OpenAI Gym). It is small enough that a plain Q-table works and learning is fast, but it has all the essential RL ingredients: states, discrete actions, sparse reward, and episodes that can end in failure.

Going block by block in strict order means each snippet is runnable on its own and builds on the last. By the final block you'll have a Q-table that has converged and an agent that crosses the lake reliably — concrete proof that the abstractions from post 3 actually compute.

The setup is intentionally minimal: Gymnasium for the environment and NumPy for the Q-table. Install with pip and import all three modules — random is needed for the epsilon-greedy coin flip. Gymnasium is actively maintained and API-compatible with the original Gym, so most older tutorials still apply with the import name swapped.

Keeping dependencies this light is itself a teaching point. Tabular Q-learning needs no deep learning framework, no GPU, nothing heavy. The entire learning algorithm is a NumPy array and a few lines of arithmetic, which makes it the ideal place to see RL's core mechanics without the distraction of neural-network machinery.

Here we create the environment and read its dimensions. FrozenLake is a 4x4 grid, so there are 16 discrete states (one per tile) and 4 discrete actions (move left, down, right, up). Setting is_slippery=False makes the ice deterministic — an action always moves where you intend — which makes learning clean and fast for a first run.

Reading n_states and n_actions from the spaces rather than hard-coding them is good practice: the same code then works if you swap in the 8x8 version or another discrete environment. The agent starts at tile 0 and must reach the goal tile, avoiding holes that end the episode with zero reward. That sparse reward structure is exactly what makes RL interesting even here.

The Q-table is the agent's entire brain: a 16-by-4 NumPy array, initialized to zeros, where Q[state][action] holds the estimated value of taking that action in that state. Starting at all zeros means the agent begins with no knowledge and no preferences — every action looks equally (un)promising until experience says otherwise.

The three hyperparameters set the learning behavior. alpha=0.8 is a high learning rate, fine for this deterministic toy. gamma=0.95 makes the agent fairly far-sighted, which matters because the only reward is at the distant goal. epsilon=1.0 starts the agent fully exploring, since with a blank Q-table greedy actions would be meaningless. We will decay epsilon during training so it gradually shifts from exploring to exploiting.

This flow diagram describes the task the agent is solving so the code has context. It begins on the start tile (state 0), must traverse frozen tiles that are safe to step on, must avoid holes that immediately end the episode with no reward, and is trying to reach the goal tile, which is the only place it earns a reward of 1.

This is a deliberately sparse reward: across an entire episode the agent gets a single bit of useful signal, and only if it succeeds. That is why the value-propagation machinery from post 3 matters so much here — the reward at the goal has to flow backward through the Q-table to make the earlier tiles look valuable, which takes many episodes.

This is the outer training loop, and it implements the explore-exploit logic directly. For each of the 2000 episodes, we reset the environment to the start state and step until the episode ends. The action choice is epsilon-greedy: with probability epsilon we sample a random action (explore), otherwise we take argmax of the Q-row for the current state (exploit the best known action).

The env.step(action) call is the heart of the interaction — it returns the next state, the reward, and flags telling us whether the episode terminated or was truncated. This block is the direct realization of the agent-environment loop from post 1, now with a concrete environment and a concrete action-selection rule wired in.

This block, which lives inside the while loop, is the actual learning — the Bellman update from post 3 in running code. best_next is the highest Q-value available in the next state, the agent's estimate of the future. target adds the immediate reward to that discounted future value. The += line nudges Q[state, action] a fraction alpha toward this target, shrinking the temporal-difference error.

After the inner loop, we decay epsilon by multiplying it by 0.999 each episode, with a floor of 0.01. This is the crucial schedule: early on the agent explores almost randomly, but by the later episodes it almost always exploits its now-accurate Q-table. Forgetting this decay is one of the classic bugs called out in post 5.

This trace annotates the four lines that do the real work, in plain language. best_next reads off the best achievable value from the next state. target combines the reward just received with that discounted future value to form a better estimate of the current action's worth. The update line moves the stored Q value toward that target by a step proportional to alpha.

The fourth line, epsilon decay, is shown because it is easy to overlook yet essential: it is what shifts the agent's behavior from exploration to exploitation over the course of training. Read together, these lines are the entire learning algorithm — everything else in the program is just plumbing to feed them experience.

Evaluation must use a different policy than training. Here we turn exploration off entirely and act purely greedily — always argmax of the Q-row — across 100 fresh episodes, counting how many reach the goal. On non-slippery FrozenLake a well-trained agent should solve it essentially every time, so you expect a success rate near 100%.

The separation matters: during training we deliberately took random actions to learn, but the policy we deploy and judge is the greedy one derived from the learned Q-table. Evaluating with exploration still on would understate the agent's true competence. This is the same train-versus-evaluate discipline you saw in supervised learning, adapted to RL's acting context.

These are the levers you turn when an RL run misbehaves, and experimenting with them is the fastest way to build intuition. alpha controls how aggressively each experience overwrites old estimates; too high and learning is noisy, too low and it crawls. gamma sets the planning horizon; too low and the agent won't value the distant goal enough to learn the path.

The epsilon decay schedule governs the explore-exploit balance over time, episodes simply buys more learning, and flipping is_slippery to True introduces stochastic dynamics that make the task genuinely hard and force a more robust policy. Change one knob at a time and watch the success rate to see what each actually does.

These are the FrozenLake-specific versions of the failure modes that post 5 covers in general. If epsilon stays high, the agent keeps acting randomly and its greedy policy never stabilizes. If gamma is too low, the distant goal reward is discounted into near-irrelevance and the agent never learns to seek it. Too few episodes leaves the Q-table essentially blank, so behavior stays random.

Forgetting to decay exploration is the single most common beginner mistake, because the code runs and the reward chart even improves slightly, masking the problem. Watching for these specific symptoms here trains the debugging instincts you will need on harder environments where the failures are less obvious.

The takeaway from this build is that RL's intimidating theory reduces to a surprisingly short, runnable program. A NumPy array, an epsilon-greedy choice, and a one-line Bellman update — repeated across episodes — is enough to learn optimal behavior in a real (if small) environment. Everything more advanced, like DQN, replaces the Q-table with a neural network but keeps this same skeleton.

Next we turn to the failure modes. This code worked cleanly because FrozenLake is forgiving; real RL projects rarely are. Post 5 catalogs the traps — reward hacking, bad exploration schedules, sparse-reward stalls, overfitting, and unsafe sampling — that turn promising agents into frustrating ones.