Playing a Neural Network's version of GTA V: GAN Theft Auto

A neural network can run an entire slice of Grand Theft Auto 5—generating the visuals, responding to key presses, and reproducing game-like physics—without relying on the game’s original rules engine. In “GAN Theft Auto,” the model takes real-time player inputs (left/right, turning into obstacles, interacting with the road) and outputs pixel frames that include not just scenery and motion, but also lighting, reflections, shadows, and collision responses. The result matters because it demonstrates a practical path from “AI that plays a game” to “AI that becomes the game,” hinting at future software where core simulation and rendering are learned rather than hand-coded.

The project builds on NVIDIA’s GameGAN research, but pushes beyond a simple GAN that maps inputs to frames. Here, the neural network is effectively the environment: it decides how the road looks, how the car turns, what happens when the vehicle hits walls or barriers, how background elements shift with movement, and how light behaves on the car’s surfaces. The team also adds super sampling on top of the model output to reduce native pixelation, including a selected “times 8” super-sampled version that produces the most playable, visually coherent results.

Getting to GTA 5 required solving data and compute bottlenecks. Manual data collection would have taken months, so Daniel Kukiela created a rules-based AI to drive through GTA 5 and generate training footage. Multiple AI agents were used simultaneously to accelerate data gathering, with cars appearing to clip or overlap in the same road segment during capture. Training itself was GPU-memory intensive; early experiments ran on RTX 8000 systems, and later work leveraged NVIDIA’s DGX station with four A180 GPUs (320 GB total VRAM) to scale experiments.

Initial models were trained on a limited highway section and struggled with hard boundaries—driving into walls could quickly confuse the system. Subsequent training introduced wall collisions, and the model learned plausible contact behavior: the car approaches, impacts, then turns and slides along the barrier rather than simply breaking down. Lighting improvements followed as well, including moving sun reflections on windows and matte surface highlights, plus shadows that track with motion.

The team also experimented with other vehicles, adding police cars and collisions. Results were mixed: sometimes the model handled interactions partially (including a collision where a police car appears to split), but often other cars would disappear or the player vehicle could not steer through them reliably. Even so, the project’s strongest takeaway is the breadth of learned dynamics—road perspective changes, background parallax, and subtle vehicle roll during turns—emerging from training on real gameplay frames.

After roughly two months of work and limited training scope, the project still produced a playable, GTA-5-recognizable environment entirely generated by a neural network. The team plans to host the base model and a super-sampler via GitHub, and invites others to train and share new variants, with the next obvious research targets being larger roaming ranges and more robust multi-car collision behavior.