Running our Reinforcement Learning Agent - Self-driving cars with Carla and Python p.5

Reinforcement learning training for a self-driving agent in CARLA is stitched into a full end-to-end loop: TensorFlow session setup, GPU memory throttling, model checkpointing, and an episode-based experience replay pipeline that updates a DQN agent while it simultaneously plays. The core outcome is that the training run can finally proceed reliably—after a long chain of fixes for shape mismatches, variable naming errors, and TensorBoard logging—so the system can start producing measurable learning signals like reward trends, Q-value changes, and loss/accuracy curves.

The training script begins by enforcing repeatability: random seeds are set for Python, NumPy, and TensorFlow. It then configures GPU behavior to avoid out-of-memory crashes by limiting per-process GPU memory fraction (the tutorial notes that running multiple agents or other GPU workloads requires this). With the session configured, the code ensures a models directory exists, instantiates the DQN agent, and launches a training thread that runs the agent’s training loop in parallel.

Once the agent is ready, the main loop runs episodes under a progress bar. Each episode starts with a CARLA environment reset to obtain the initial observation (the “current state”). The loop uses an epsilon-greedy policy: with probability tied to epsilon, the agent picks a random action (fast, avoids neural-network inference), otherwise it selects an action based on predicted Q-values from the network. A key practical detail is timing: the code sleeps based on a target FPS so that both random-action steps and model-action steps occur at roughly comparable rates—important because the effective frame rate influences training stability.

For every step, the environment returns a new state, a reward, and a done flag. The transition (current state, action, reward, new state, done) is pushed into replay memory via an update call. The script increments step counters and breaks out cleanly when the episode ends. Statistics are tracked across episodes—minimum, maximum, and average reward—and the model is saved when performance crosses a chosen threshold (the notes suggest average reward as a better criterion than minimum reward, since minimum can be dragged down by unavoidable early crashes).

A major portion of the transcript is troubleshooting so training can actually run: input-shape errors, threading/name mismatches, missing TensorBoard callback imports, and CARLA state/image dimension bugs (including case-sensitive “height/width” variable fixes). TensorBoard is handled carefully because reinforcement learning can call fit repeatedly; a custom TensorBoard setup logs per episode rather than per frame to prevent log-file explosion.

Results are mixed but informative. TensorBoard curves show accuracy spikes to 100% at times (flagged as suspicious), while loss sometimes explodes to extremely large values—an indicator that something is unstable in learning. Even so, qualitative behavior improves in certain runs: the agent can stay in a lane and make turns, and Q-values visibly change as it interacts with the environment. The training speed is also reported (e.g., around ~18.5 FPS in one run), reinforcing that performance constraints matter.

The next steps are framed around controlled experimentation: train longer before declaring changes effective, adjust network architecture, and test environment/action-space modifications such as giving the agent throttle and brake control. There’s also interest in generalizing beyond CARLA’s current setup—potentially moving toward Grand Theft Auto 5—but only if camera position and sensor parameters (like field of view) can be made dynamic so the model doesn’t collapse when inputs shift.