Social learning in independent multi-agent reinfor… | Kamal N’dousse | OpenAI Scholars Demo Day 2020

Social learning is central to human intelligence, but it’s unclear when independent reinforcement-learning agents can benefit from one another’s behavior. Kamal N’dousse frames the core problem as a test of whether RL agents can learn from “experts” merely by observing them in shared environments—and if so, under what conditions that social signal beats learning from direct experience.

The talk starts with a human-scale motivation and a monkey parable from experimental sociology: monkeys punish any peer that tries to reach bananas after prior punishment (cold water). Even when the punishment stops, the behavior persists, and new monkeys quickly get punished for attempting the same forbidden action—suggesting a cultural pattern can emerge from observation and enforcement. N’dousse notes the story is apocryphal, but treats it as a template for studying social learning in artificial agents.

To investigate, he builds tools for independent multi-agent RL. He develops marl grid, an open-source grid-world suite compatible with the OpenAI Gym API, designed to scale to many agents and support reproducible experiments. A key environment is “goal cycle,” where agents must traverse gold tiles in a specific order to earn reward; stepping on tiles out of order triggers a configurable penalty. By tuning the penalty, the environment changes how costly exploration is: with low penalties, agents can “mess up” while still eventually finding rewards, while with high penalties, exploration becomes so aversive that agents learn to commit early to a first successful path and avoid alternatives. This knob lets him control the difficulty of learning from the environment itself—crucial for testing whether expert demonstrations provide an advantage.

On the algorithm side, he reports that DQN struggled when long-horizon behavior required memory, even after adding LSTM and trying limited prioritized experience replay. Switching to PPO produced a major improvement. Further gains came from a practical technique he calls “hidden state refreshing”: during PPO’s repeated update steps, the LSTM hidden states are periodically refreshed so the agent’s internal memory doesn’t become stale relative to the evolving policy. With this change, goal-cycle agents achieve higher rewards and more stable training.

With the infrastructure in place, N’dousse turns to observational learning. He first tries to replicate a DeepMind result, “Observational Learning by Reinforcement Learning” (where experts are hard-coded and novices learn via RL in a simple grid). In that earlier work, experts speed up novice learning, but the novices ultimately don’t outperform what they would achieve learning alone. In N’dousse’s replication on a cluttered grid with a single goal, experts again fail to provide a learning speedup.

The more interesting outcome appears in goal cycle. When the cycle structure is masked from novice agents—so direct environmental information is less immediately actionable—novices can learn to follow expert agents’ behavior. In demonstrations, novices converge toward the experts’ strategy, though they may land at slightly lower performance when an expert happens to get trapped, indicating imitation can propagate suboptimal trajectories.

The emerging conclusion is cautious: learning from experts is difficult when agents can already infer the task from direct interaction. Social cues become valuable when the environment makes the right strategy hard to discover on one’s own, and when expert behavior provides information that direct observation doesn’t readily reveal. Next steps include scaling to more goals and varying penalty settings, plus measuring whether novices truly acquire the same skill by testing them in new environments without experts. He also proposes exploring priors or mechanisms that encourage social learning, and asks whether novices could ever surpass experts—an open direction for future experiments.