Large-Scale Study of Curiosity-Driven Learning

This paper asks a simple but largely unanswered question in reinforcement learning (RL): can an agent learn useful behavior when it receives only intrinsic reward from curiosity—specifically, a prediction-error (surprisal) signal—without any extrinsic reward or end-of-episode (“done”) signal? The question matters because most RL success depends on carefully engineered external rewards that are dense and well-shaped, yet designing such rewards does not scale to new environments. If curiosity alone can drive competent exploration and skill acquisition, then reward engineering could be reduced to selecting environments rather than specifying detailed reward functions.

The authors conduct what they describe as the first large-scale empirical study of purely curiosity-driven learning across 54 standard benchmark environments, spanning Atari games, Super Mario Bros., Unity-based 3D navigation/maze tasks, Roboschool physics tasks, and two-player Pong. The broader significance is twofold. First, it provides evidence that intrinsic objectives can substitute for extrinsic reward in many human-designed environments, suggesting that extrinsic scoring systems may often correlate with “novelty-seeking” behavior. Second, it investigates how the curiosity mechanism depends on representation learning: curiosity requires learning a forward dynamics model in some feature space, and the choice of feature space affects both learning stability and generalization.

Methodologically, the paper uses a dynamics-based curiosity formulation. For an agent observing state (or observation) $x_t$, taking action $a_t$, and transitioning to $x_{t+1}$, they embed observations via $\varphi (x)$ and train a forward dynamics model to predict the next embedding $\varphi (x_{t+1})$ conditioned on $x_t$ and $a_t$. The intrinsic reward is defined as prediction error/surprisal, implemented as a mean-squared error corresponding to a fixed-variance Gaussian density: $r_t=-\log p(\varphi (x_{t+1})\mid x_t,a_t)$, operationalized as $\Vert f(x_t,a_t)-\varphi (x_{t+1}){\Vert }_2^2$ (up to constants). The policy and curiosity dynamics are trained jointly using PPO (Proximal Policy Optimization), with multiple practical stabilizers: reward normalization by a running estimate of the standard deviation of discounted returns, advantage normalization, observation normalization from a short random rollout, and increased parallelism (typically 128 parallel actors; 2048 for the large-scale Mario run). They also note that they remove the “done” signal in Atari to avoid reward leakage through episode termination.

A central experimental axis is the feature space $\varphi$ used for the forward model. They compare: (1) pixels ($\varphi (x)=x$), (2) random features (a fixed, randomly initialized convolutional embedding network), (3) inverse dynamics features (IDF; features learned by training an embedding network to predict the action from consecutive states), and (4) VAE features (features from a variational autoencoder latent mean). Their qualitative and quantitative results show that pixels perform poorly, while random features and IDF are strong baselines. Specifically, on 8 representative Atari games (Figure 2), raw pixels “does not work well across any environment,” VAE is “either same or worse than random and inverse dynamics,” and IDF is better than random features in 55% of Atari games. Across the full Atari suite, they report that an IDF-curious agent collects more extrinsic reward than a random agent in 75% of games, while an RF-curious agent does so in 70%; IDF beats RF in 55% of games. Importantly, these extrinsic returns are used only for evaluation, not training, so the intrinsic objective is still the only learning signal.

The paper also studies generalization. In Super Mario Bros., they pre-train on Level 1-1 using curiosity only, then fine-tune on novel levels (Level 1-2 and Level 1-3) using curiosity only. They find strong transfer to Level 1-2 for both RF and IDF, but weaker transfer to Level 1-3, which differs more drastically (a day-to-night color scheme shift). Their interpretation is that learned IDF features generalize better when the visual distribution shifts substantially, while random features may suffice when global statistics match.

They further explore how training stability and compute affect outcomes. In a large-scale Mario experiment, they increase the number of parallel environment threads from 128 to 2048. The result is a substantial performance jump: with the larger batch/parallelism, the agent discovers 11 different Mario levels, finds secret rooms, and defeats bosses, indicating that curiosity-driven learning benefits from stronger optimization stability in the underlying PPO training.

Beyond pure exploration, the authors test whether curiosity helps when extrinsic reward is sparse or terminal. In a Unity 3D maze with 9 rooms and only a terminal reward of $+1$ at the goal, extrinsic-only PPO never finds the goal in all trials (so it receives no meaningful learning gradient), while PPO with intrinsic curiosity converges to reaching the goal consistently. They also report preliminary Atari experiments on sparse-reward games: in 4 out of 5 selected games, adding curiosity improves performance (details deferred to an appendix table). They emphasize that these reward-combination experiments are not the paper’s main focus and use a fixed intrinsic coefficient (0.01) without tuning.

Finally, they highlight a key limitation of prediction-error curiosity in stochastic settings. If the environment’s unpredictability is driven by stochasticity that the agent cannot control (or that the agent itself can induce), then the agent can be attracted to entropy rather than progress. They demonstrate this with a “noisy-TV” style manipulation: adding a local source of randomness (a TV that changes channels) to the Unity maze. They observe that curiosity learning slows dramatically when such stochasticity is present, and while agents may eventually recover and learn the extrinsic goal given enough time, the attraction to noise is a real failure mode.

Limitations are implied by the methodology and explicitly discussed in the discussion section. The approach relies on learning a forward dynamics model; if the model class is insufficient, the observation is partially observable, or the environment is stochastic in ways that create high entropy transitions, prediction-error rewards can misalign with task progress. Additionally, feature learning methods like VAE and IDF introduce non-stationarity because the embedding changes as auxiliary models train; the authors mitigate this by using random features (stable) and by carefully normalizing inputs and rewards. Another limitation is that the study is empirical and benchmark-focused: while 54 environments are diverse, the results may not generalize to environments with different reward structures, observation modalities, or stochasticity patterns.

Practically, the paper suggests that curiosity-driven exploration can be used as a default intrinsic objective when reward engineering is expensive, and that many standard RL benchmarks may already be “curriculum-like” in ways that align with novelty-seeking. Researchers and practitioners in RL should care because this work provides (i) a scalable recipe for reward-free training using PPO plus dynamics-based curiosity, (ii) evidence that random-feature curiosity is a surprisingly strong and stable baseline, and (iii) concrete warnings about stochasticity-driven exploitation. Teams building agents for new domains may use curiosity pretraining on unlabeled environments and then fine-tune with sparse extrinsic signals, potentially reducing the need for dense reward shaping.