Generative Modelling | Sadhika Malladi | 2018 Summer Intern Open House

Generative modeling is positioned as a practical way to learn the underlying structure of data—so a model trained on one set of examples (like dogs) can produce plausible, previously unseen samples. That same idea becomes especially valuable in reinforcement learning, where agents need to understand how actions change the world. Instead of learning purely from trial-and-error, “world modeling” aims to build an internal model of environment dynamics so an agent can plan and generalize with far less data—an outcome that matters because collecting experience in games or real systems is expensive.

In the world-modeling setup, the goal is to learn basic movement skills first—such as stepping forward, turning without falling, or reversing—so later tasks don’t require new, highly specific instructions. A concrete example uses a simple car racing game: the agent learns how to accelerate, slow down, and turn, then can handle a track it has never seen before by relying on learned interactions between its actions and the environment. The promise is sample-efficient, transferable reinforcement learning. But current model-based reinforcement learning often underperforms model-free methods because learned world models can be brittle: small reconstruction errors compound during rollout, creating artifacts that the model can’t correct once they start propagating.

A key technical issue is how frames are represented. The car racing model uses a variational autoencoder (VAE): frames are encoded through convolutional layers into latent variables (Z), decoded back into reconstructions, and trained by comparing reconstructions to the original frames. Those latents are also fed into an RNN to track time-dependent state, such as where obstacles are and what the game situation looks like. The approach works well for car racing because the environment has relatively few elements—mostly track geometry and a simple highway structure—so the latent space can focus on meaningful, task-relevant factors.

The same approach struggles on Pong. After encoding and reconstructing, the model fails to preserve crucial objects like the ball and the other paddle, making it hard for the agent to learn where key elements are. The proposed fix is to pair the VAE-style encoder with a stronger decoder—specifically a PixelCNN conditioned on the time state—so latents are forced to represent obstacle-related information rather than generic background. With that conditioning, latents become more “time dependent,” and PixelCNN’s ability to generate high-resolution images is expected to improve generalization to richer games.

The discussion then shifts from world modeling to invertible flow models, which are highlighted for expressive latent spaces and efficient sampling. By learning a transformation between a simple prior distribution (random noise) and data (faces, cats, dogs), these models can generate realistic outputs by sampling noise and mapping it through an invertible network. Training and architecture rely on maximizing likelihood via the log-determinant of the transformation, but invertibility constraints reduce expressivity and can make training finicky. Architectures such as RealNVP and its refined variant (GLOW) use multi-scale designs, squeezing operations, normalization convolutions, and coupling layers to manage these constraints. A practical improvement described is parameter sharing across coupling layers, cutting memory use dramatically while maintaining sample quality (measured via bits-per-dimension) and improving speed. The work also notes that invertible models can fail when they enter regions where invertibility breaks down, motivating ongoing efforts to improve flow model training and expressivity.