Continuous control with deep reinforcement learning

This paper asks whether deep reinforcement learning for continuous-action control can be made stable and effective without discretizing actions, and whether it can scale to high-dimensional physical control tasks using the same core learning algorithm and hyperparameters. This matters because many real control problems—robot locomotion, manipulation, and driving—require real-valued, multi-degree-of-freedom actions. Prior deep RL success (notably DQN) relied on discrete actions and on computing argmax over actions at every step, which becomes impractical in continuous domains.

The authors’ central contribution is an actor-critic, model-free algorithm they call Deep Deterministic Policy Gradient (DDPG). It adapts the deterministic policy gradient (DPG) idea to continuous action spaces, while importing two key stability mechanisms from DQN: (1) off-policy learning with a replay buffer to reduce sample correlations, and (2) target networks to stabilize temporal-difference targets. They further add batch normalization to handle differing feature scales across tasks and use temporally correlated exploration noise (Ornstein–Uhlenbeck) suited to physical systems with inertia.

Methodologically, the study is an empirical evaluation across a suite of simulated physics environments implemented in MuJoCo. The environments span classic control (cartpole variants, pendulum, cart) and more challenging multi-joint manipulation and contact-rich tasks (e.g., gripper, puck striking/canada, locomotion such as cheetah, hopper, hyq, walker2d), plus a non-physics baseline environment (TORCS). For each task, the authors train DDPG using both low-dimensional state inputs (e.g., joint angles/positions) and high-dimensional pixel observations from a fixed camera. For pixel inputs, they use action repeats: each chosen action is applied for 3 simulation steps while rendering each time, producing stacked RGB frames (downsampled to 64×64) so the agent can infer velocities from frame differences.

The algorithmic details are standard for DDPG: a deterministic actor (s|${\theta }^{\mu }$) outputs continuous actions, and a critic approximates the action-value function $Q(s,a|{\theta }^Q)$. The critic is trained off-policy by minimizing a mean-squared Bellman error over minibatches sampled uniformly from a replay buffer of size $10^6$. The target value uses slowly updated target networks $Q^{\prime }$ and ${\mu }^{\prime }$ with soft updates ${\theta }^{\prime }\leftarrow \tau \theta +(1-\tau ){\theta }^{\prime }$ where $\tau =0.001$. The actor is updated using the deterministic policy gradient via the chain rule, using ${\nabla }_{a}Q(s,a)$ evaluated at $a=\mu (s)$. Optimization uses Adam with learning rates $10^{-4}$ for the actor and $10^{-3}$ for the critic; the discount factor is $\gamma =0.99$; the critic uses $L_2$ weight decay of $10^{-2}$. Network architectures differ for low-dimensional vs pixel inputs: low-dimensional models use two hidden layers (400 and 300 units), while pixel models use three convolutional layers followed by two fully connected layers (200 units each). Actions are bounded using a $\tanh$ output layer. Training is run for at most 2.5 million environment steps per task; results are averaged over 5 replicas.

The paper’s key findings are that DDPG robustly solves more than 20 simulated physics tasks, often reaching performance competitive with (and sometimes exceeding) a strong model-based planning baseline (iLQG) that has full access to the simulator dynamics and derivatives. The authors report normalized returns where a naive random policy has mean score 0 and iLQG has mean score 1 (except TORCS, which uses raw reward). Across tasks, DDPG’s average and best observed scores are frequently positive and often near or above 1.

For example, in the cartpole swing-up task, DDPG achieves $R_{av,lowd}=0.844$ and $R_{best,lowd}=1.115$, while from pixels it achieves $R_{av,pix}=0.482$ and $R_{best,pix}=1.138$. In cartpoleBalance, DDPG reaches $R_{av,lowd}=0.951$ and $R_{best,lowd}=1.000$, and from pixels $R_{av,pix}=0.335$ and $R_{best,pix}=0.996$. In the harder hardCheetah task, DDPG attains $R_{av,lowd}=1.311$ and $R_{best,lowd}=1.990$, and from pixels $R_{av,pix}=1.204$ and $R_{best,pix}=1.431$, while the original DPG baseline with replay buffer and batch normalization (labeled $cntrl$) has $R_{av,cntrl}=-0.031$ and $R_{best,cntrl}=1.411$. In locomotion walker2d, DDPG achieves $R_{av,lowd}=0.705$, $R_{best,lowd}=1.573$, and from pixels $R_{av,pix}=0.944$, $R_{best,pix}=1.476$. In manipulation-like tasks, performance is also strong: for gripper, DDPG yields $R_{av,lowd}=0.655$, $R_{best,lowd}=0.972$, and from pixels $R_{av,pix}=0.406$, $R_{best,pix}=0.790$. For fixedReacherSingle, DDPG reaches $R_{av,lowd}=0.954$, $R_{best,lowd}=1.000$, and from pixels $R_{av,pix}=0.827$, $R_{best,pix}=0.995$. The paper also includes ablation-style observations: removing the target network or batch normalization leads to very poor learning in many environments, and both additions are necessary for consistent performance.

The authors also examine critic value accuracy by comparing learned $Q$ estimates after training to true returns on test episodes. They report that in simpler tasks, DDPG’s value estimates are accurate without systematic bias; in harder tasks, estimates are worse, but the policy learning still succeeds.

A notable qualitative claim is that DDPG can learn end-to-end from raw pixels in many tasks, sometimes as fast as learning from low-dimensional state, plausibly due to action repeats simplifying dynamics and convolutional layers producing useful representations.

Limitations are acknowledged in the conclusion and are also implicit in the methodology. Like many model-free RL methods, DDPG requires a large number of training episodes/interaction steps to find solutions. The paper also notes that theoretical convergence guarantees are lost when using nonlinear function approximators. Additionally, performance variability across replicas is evident in the table (some tasks have low average but high best scores), suggesting sensitivity to initialization and exploration. Finally, while DDPG is compared to a strong planning baseline, the evaluation is in simulation; transfer to real robotics is not demonstrated here.

Practically, the results suggest that a relatively simple actor-critic framework—deterministic policy gradient plus replay buffer, target networks, and batch normalization—can serve as a general-purpose backbone for continuous control, including vision-based control. Researchers and engineers working on robotics, autonomous driving, and simulated manipulation should care because the paper provides a recipe that scales across many tasks with largely unchanged hyperparameters and architectures, and because it demonstrates that end-to-end pixel-to-action learning is feasible in a wide range of continuous control settings.

Overall, the paper’s core message is that stability techniques from deep Q-learning can be successfully transplanted into deterministic actor-critic learning for continuous actions, enabling robust performance across diverse high-dimensional control problems, with competitive results relative to model-based planners and with meaningful success from raw visual inputs.