A. I. Learns to Play Starcraft 2 (Reinforcement Learning)

A reinforcement-learning agent can learn to play StarCraft 2 at least at the “macro” level by using a custom, simplified minimap representation as input and a small set of high-level actions—then training with a reward that favors sustained combat rather than resource hoarding or endless survival. After starting from a baseline where random action selection never wins, the training setup produced a model that reached roughly a 70% win rate against the hard computer bot, with peak training reward around 200. The result matters because it shows a workable path from game state to RL signals without needing full control of every in-game micro decision.

The approach begins by reframing StarCraft 2’s complexity into inputs, outputs, and reward. Instead of feeding raw video frames, the agent receives a “mini-map” built from a blank canvas: minerals, gas, building health, enemy starting location, enemy units, enemy structures, the player’s nexus and other structures, vespene, and unit positions. Visibility is encoded by brightness—minerals not currently observed by units appear dim—so the agent can infer uncertainty. Attack units (void rays) are colored distinctly to make combat-relevant information easier to parse.

On the action side, the agent is not controlling every unit command. It chooses among six macro actions implemented as explicit rules: (0) expand by ensuring supply, probes, assimilators, and a new base; (1) build stargates by creating required tech structures (gateways and cybernetics cores) and then stargates (currently capped at one per base); (2) build a void ray when affordable; (3) scout by sending a probe infrequently (throttled to once every 200 frames to avoid constant “suicide scouting”); (4) attack by prioritizing nearby units and buildings, otherwise moving void rays toward the enemy start location; and (5) retreat/fall back by returning void rays to base after fights.

The training hinges on reward design, where naive choices can derail learning. A benchmark run of 200 games using random actions produced zero wins, confirming the task is nontrivial. For learning, the system avoids sparse “win/loss only” rewards because the agent faces thousands of steps per game and can’t easily assign credit. Intermediate rewards are used instead: the agent earns a small per-step reward for each void ray actively in combat, while a terminal reward (and punishment) is applied for winning or losing. Attempts to reward resource gathering or total army size tended to encourage stalling—lengthening games or building without finishing—so combat-focused reward proved more effective.

Finally, the implementation connects StarCraft 2 to Stable Baselines 3 (using PPO) through a custom OpenAI Gym environment. Because the game loop and the RL loop can’t communicate directly in a clean way, the system relies on a shared state file (pickle) and multiple processes: one waits for actions from the RL model, another executes commands in StarCraft 2, redraws the minimap, computes rewards, and signals episode completion. After training, the best model achieved about a 70% win rate versus the hard bot, marking a strong early milestone.

The next planned step is to improve beyond macro decisions—especially micro-level targeting and movement. Experiments with “hunting” void rays at random coordinates when no enemies are visible performed worse, suggesting that coordinate selection should likely become its own learned micro policy rather than hand-coded logic. The project is positioned as a foundation for combining macro RL with a separate micro RL module later.