Drone vs. Zombies: Reinforcement Learning

The idea

Two of us, one game. My teammate had built DEAD AIR, a drone-versus-zombies game in Godot. I built the brain: an agent that flies the drone with realistic quadcopter physics, finds its way through a procedurally generated town and takes out zombies, without anyone telling it how.

How it learns

• Real flight physics: the agent controls a simulated 5-inch FPV quad the way an acro pilot does, through thrust and rotation rates. The game's flight model matches the simulation to within 2 cm over 11 m.
• What it senses: an IMU, a 72-beam lidar, a forward depth grid, a camera-style zombie tracker and incoming projectiles, 579 inputs in total.
• Training: PPO and recurrent PPO on 12 parallel workers, through a curriculum that goes from hovering to navigating to hunting.
• Reward design: progress is measured along the shortest open path, so the agent is never rewarded for flying into a wall. Crashes, jerky control and overshooting waypoints cost points. It is pure reinforcement learning, with no safety shield and no imitation learning.

Results

On test maps the agent had never seen:

• Hovering: 99.5% success after 16 minutes of training.
• Combat: 81% of rounds cleared, 93% of zombies hit and 47% shot accuracy.
• Navigation is the hard part. After 154 million steps the agent beats a hand-written autopilot, but tunnels are still unsolved. Moving zombies and return fire come next.

The videos below are recordings of the trained agent on unseen maps.

Built with Python, Gymnasium, PyFlyt, Stable-Baselines3, PyTorch and Godot.

What I learned

Reward design is everything. Most of my time went into finding out what the agent was really being rewarded for, because it always finds the loophole before it finds the behaviour you wanted. Measuring every stage on held-out maps kept me honest about what actually worked.