Stabilizing Q-Learning in CartPole
Introduction If you are interested in reinforcement learning, you have probably heard of CartPole [6], often considered the “Hello World” environment of the field. This is partly because the task is simple to understand, and partly because each episode ends quickly, allowing you to observe learning progress within a short time. Figure 1. CartPole environment visualization Looking back, I have run many experiments in CartPole before, but not in a structured or systematic way, which caused me to forget most of the details over time. In this post, I aim to consolidate those trials into a coherent narrative, focusing in particular on why Q-learning can become unstable and how techniques such as experience replay, Double DQN, and failure-focused sampling can improve robustness. ...