Robot Dog Learns to Walk - Bittle Reinforcement Learning p.3

Reinforcement learning for Boston Dynamics–style quadruped locomotion is finally producing usable walking gaits in NVIDIA Isaac Sim—but only after a major pivot from camera-based perception to fast IMU sensing and a careful redesign of the action space. The core breakthrough is that the Bittle robot’s training becomes practical once observations come from an inertial measurement unit (IMU) rather than a camera stream, because camera frame capture in the simulator throttles training to roughly seven steps per second. At that speed, reaching the tens of millions of steps typically needed for a well-trained policy becomes measured in weeks.

With the IMU wired into the Isaac Sim agent, the project shifts toward learning control policies that can stabilize the body and reduce jitter. The next bottleneck is reward shaping: early experiments that fail to penalize rapid direction changes lead to “local optimum” behaviors—gaits that look like they exploit quirks of the physics and are effectively trapped there. The most effective fix is a movement-direction-change penalty: every servo direction change within a one-second window (implemented as a hyperparameter tied to 20 steps at 20 Hz) incurs a squared penalty, strongly discouraging jitter while still allowing forward progress.

On the control side, the biggest performance jump comes from using Discrete Delta PPO (DDPPO) rather than standard continuous-control methods like SAC, TD3, or continuous PPO. The motivation is practical: servo control is continuous in theory, but learning over thousands of possible motor positions is hard. Discrete delta reframes the problem as choosing a small relative adjustment from a small set of bins (e.g., deltas like −0.3, −0.1, −0.03, and their positive counterparts). Instead of selecting an absolute position among an enormous range, the policy picks a delta that gets added to the current position—turning an effectively infinite action space into a manageable discrete one.

After extensive tuning, the best-performing configuration reported is DDPPO with a 20 Hz action rate, a seven-bin delta set, and a seven-frame observation stack. Multiple algorithms and refresh rates were tested (including 60, 30, 20, 10, and down to 10 Hz), but DDPPO stands out as the fastest path to stable walking. Training trajectories show that some reward designs produce walkers that are reliable but not visually ideal—often keeping a low center of gravity or alternating “move-stop” patterns.

To further improve gait stability, the reward function is extended with a z-axis bounce penalty computed as the standard deviation of recent z history (history length 10). The result is a more walker-like, steadier policy that resembles earlier stable behaviors before the agent learned to hop. Even with these gains, the work remains iterative: every new restriction risks unintended side effects, and the presenter notes that longer training might naturally smooth motion without extra penalties. The overall direction is clear: fast IMU observations plus discrete delta control plus targeted jitter and bounce penalties are the recipe that turns simulation into a workable training loop for quadruped locomotion.