Reinforcement Learning with Stable Baselines 3 - Introduction (P.1)

Stable Baselines 3 is positioned as a shortcut for reinforcement learning: it standardizes the workflow so people can swap algorithms quickly while keeping the environment—and its reward/observations—separate. The core promise is time savings and reduced engineering pain: once an environment is defined, training different RL methods becomes mostly a matter of changing one line of code, rather than rewriting the full training loop each time.

The tutorial starts by mapping the moving parts of reinforcement learning. An environment is the task (for example, LunarLander-v2), a model is the trained algorithm instance, and an agent is the decision-maker that interacts with the environment. Each interaction happens in a loop: the agent observes the current state (observations/states), chooses an action, the environment advances one step, and then returns a new observation plus a reward and a done signal. The action space can be discrete or continuous. Discrete actions are categorical choices (like left vs. right in CartPole). Continuous actions are real-valued ranges (common in robotics, where control signals like torques or positions vary continuously), and the tutorial notes that robotics often gets treated as continuous even when the underlying actuator positions are effectively discretized into many possibilities.

On the tooling side, the setup centers on PyTorch as the backend, Stable Baselines 3 for the RL algorithms, and OpenAI Gym for environments. For the example environment, Box2D is installed to support LunarLander. With Gym and Stable Baselines 3 installed, the tutorial demonstrates how to create an environment, reset it, sample an action from the action space, inspect the observation space shape, and take steps using random actions. LunarLander-v2 is used to make these concepts concrete: the observation space is a flat vector of eight values, and stepping the environment yields rewards that eventually reflect failure (e.g., a large negative reward when the lander crashes).

The training section then turns these abstractions into code-level workflow. The key decision for algorithm selection is whether the environment’s action space is discrete or continuous (and sometimes multi-discrete/multi-binary). Using Stable Baselines 3’s supported-algorithm table as a guide, the tutorial picks A2C first for a discrete setting. It trains for 10,000 time steps, then evaluates performance via metrics like episode length and average episode reward. The results are described as underwhelming—better than random, but still weak.

To test whether more training helps, the tutorial increases training to 100,000 steps, but performance does not steadily improve; episode reward trends can degrade after a point. It then swaps algorithms to PPO (again by changing the import and the model class) and retrains for 100,000 steps. PPO shows some improvements—such as staying alive longer and training faster—but both A2C and PPO are still far from strong performance at these step counts. The tutorial attributes the poor outcomes to insufficient training duration and emphasizes that real RL runs often require millions to tens or hundreds of millions of steps.

Finally, the tutorial flags the next steps: saving/loading models and tracking progress over time (e.g., with TensorBoard). The takeaway is that Stable Baselines 3 reduces friction around algorithms, but the biggest engineering challenge—and the biggest performance lever—remains the environment design, reward shaping, and observation structure, which will be tackled in later parts of the series.