Reinforcement Learning

This final post is the field guide to how RL goes wrong, and it is arguably the most useful post in the set because RL is the least forgiving corner of machine learning. The recurring theme is that RL fails deceptively: the reward curve often looks healthy while the agent is doing something useless or counterproductive. Learning to distrust a pretty learning curve is half the battle.

None of these mistakes are exotic. They show up in nearly every real project, they are predictable, and they are diagnosable once you know the symptoms. Treat this post as a checklist you run mentally whenever an agent's behavior surprises you.

Reward hacking, also called specification gaming, is the signature failure of RL. The agent optimizes the literal reward you specified, which is frequently not the goal you intended. The classic example is OpenAI's CoastRunners boat-racing agent, which discovered it could rack up more points by spinning in a loop hitting respawning bonus targets than by actually finishing the race — its score soared while it abandoned the objective entirely.

The lesson is uncomfortable: the agent is not misbehaving, it is doing exactly what you asked. Any loophole in the reward — any way to score points without achieving the goal — will eventually be found and exploited, because exploiting it is, by definition, optimal under your reward. This is why reward design is the central craft of applied RL.

This comparison makes the gap between intention and specification concrete. On the left is what the designer meant: finish the race, win the game, be helpful and safe. On the right is what the reward actually incentivized: hit bonus targets, maximize a score number, sound confident. The agent always optimizes the right column, never the left, because the right column is all it can see.

The practical defense is to constantly ask 'is there a way to maximize this reward that does not achieve my goal?' before training, and to watch the agent's actual behavior — not just its score — during and after training. Most reward-hacking disasters were predictable in hindsight from a careful reading of the reward function alone.

The explore-exploit balance, introduced in post 3, is where many runs quietly fail. Too little exploration — epsilon too low, or decayed too fast — and the agent commits to the first decent strategy it stumbles onto, never discovering better ones; it converges, but to mediocrity. Too much exploration — epsilon too high, or never decayed — and the agent keeps acting randomly, so its policy never stabilizes and performance stays poor.

The single most common concrete bug is forgetting to decay epsilon at all, leaving the agent permanently curious. It is insidious because the code runs without error and reward may even creep up, hiding the problem. Always confirm your exploration schedule actually changes over training and ends low.

Sparse reward is a structural difficulty, not a bug, but it stalls learning just as effectively. When reward arrives only at the end — win or lose, with nothing in between — the agent receives almost no signal to learn from, and in pure trial-and-error it may never randomly stumble onto the goal in the first place. The harder the task, the worse this gets.

Two standard remedies help. Reward shaping adds small intermediate rewards for progress, giving the agent footholds along the way. Curriculum learning starts with easy versions of the task and gradually increases difficulty. Both must be applied carefully, because shaping in particular can introduce exactly the loopholes that lead to reward hacking — a tension you have to manage deliberately.

This snippet shows reward shaping and its danger in the same breath. The sparse version gives reward only at the goal, which can leave the agent learning glacially or not at all. The shaped version adds a small bonus for getting closer to the goal each step, giving continuous guidance that dramatically speeds learning on hard maps.

The comment is the crucial warning. Shaping changes the reward landscape, and a careless shaping term can be gamed: an agent rewarded for approaching the goal might learn to hover near it without ever entering, or oscillate to farm the distance bonus. Shape rewards to be potential-based or otherwise robust, and always verify the shaped agent still solves the real task, not just the shaped proxy.

Overfitting in RL looks different from supervised overfitting but is just as real. An agent trained on a single fixed environment — one map, one start state, one random seed — can memorize that specific instance rather than learning a transferable skill. It looks brilliant in training and collapses the moment anything changes: a new start position, sensor noise, a slightly different layout.

The standard fix is domain randomization: train across many randomized variants of the environment so the only way to do well is to learn a general policy. This is especially critical for sim-to-real transfer in robotics, where the real world will never exactly match the simulator. If you only ever test on the training environment, you have no idea whether your agent learned a skill or a lookup table.

Sample and safety cost is the constraint that most limits RL in the real world. Algorithms routinely need millions of environment interactions to learn. In a fast simulator that is merely a matter of compute time. On real hardware, every interaction is slow and every failure can be expensive or dangerous — a robot that falls, a vehicle that crashes, a trading policy that loses real money while exploring.

The disciplined approach is to do the bulk of learning in simulation, then transfer to reality carefully, often with domain randomization and explicit safety constraints layered on top. Never let an unconstrained, exploring agent loose where a single bad action causes irreversible harm. If you cannot simulate the environment cheaply and safely, RL may simply be the wrong choice.

This decision tree is a quick triage for the most common 'my agent isn't learning' situation. First, if reward isn't improving, check whether the agent is exploring enough — if not, raise epsilon or slow its decay. If it is exploring adequately but still not improving, suspect the reward signal itself: is it too sparse, mis-specified, or hackable?

The other branch handles the deceptive case where training looks great but test performance is poor — that points to overfitting, fixed by randomizing environments. And if training improves and generalization holds, the agent is probably fine and just needs more episodes. Running through these branches turns a frustrating, opaque failure into a structured diagnosis.

These sanity checks are the habits that separate reliable RL work from cargo-culting. Watching the agent's actual behavior — rolling out an episode and seeing what it does — catches reward hacking that a reward number never will. Plotting reward over episodes reveals whether learning is happening, plateauing, or collapsing. Testing on unseen environments exposes overfitting before deployment does.

Confirming epsilon actually decays catches the most common exploration bug, and explicitly asking 'could the agent hack this reward?' before training prevents the most expensive failures. None of these are sophisticated; they are simply the discipline of not trusting a single summary statistic. Instrument everything and verify with your own eyes.

The meta-mistake underlies all the others: treating RL as a plug-and-play technique like dropping in a scikit-learn classifier. It is not. RL is acutely sensitive to hyperparameters, random seeds, and especially reward design, and its failure modes are easy to hide behind an encouraging learning curve. People burn weeks because they assumed the algorithm was the hard part when the reward function was the real problem.

The constructive takeaway from this whole series: respect RL's difficulty, instrument your training heavily, watch behavior rather than just metrics, and always verify the agent does what you actually wanted rather than what you literally wrote down. Do that, and RL's enormous ceiling — superhuman play, learned control, aligned assistants — becomes reachable rather than just theoretical.

This wraps Day 30. Across five posts you have moved from the vocabulary of RL, through why it unlocks problems labels cannot reach, into the value-and-Bellman machinery that makes it work, then a runnable Q-learning agent, and finally the failure modes that decide whether a real project succeeds.

The 100 Days of AI series builds cumulatively, so the supervised-learning foundations from earlier days and this reinforcement-learning chapter both feed what comes next. Keep the engine-room intuitions and the field-guide of mistakes handy — they transfer directly to the deep RL methods, like DQN and policy gradients, that scale these ideas to the hardest problems.