为何Deep Q-learning、A3C等算法无法学会玩Asteroids游戏？

Great question—this is one of those head-scratching cases in reinforcement learning that defies initial expectations. Let's start by laying out the weirdness of the situation:

• Tried-and-true DRL approaches like Deep Q-learning, A3C, even strategies evolved using genetic algorithms either can't learn to play Asteroids at all, or perform way worse than even casual human players.
• Unlike Montezuma's Revenge (the classic example of a sparse-reward Atari game that stumps most RL systems), Asteroids gives you a clear, immediate reward every time you hit an asteroid. So sparse rewards aren't the culprit here.

So why are these powerful DRL methods falling flat? Multiple papers have documented this underwhelming performance, and the root issues come down to a few key challenges that standard DRL pipelines aren't great at handling:

• Hidden state tracking: Asteroids lets asteroids move off-screen and re-enter from the opposite edge. A typical CNN-based DRL agent only sees the current frame, so it can't keep track of where those off-screen asteroids are or when they'll come back. Humans do this intuitively with mental models, but without explicit memory components (like LSTMs), DRL agents miss this critical context.
• Long-term strategy gaps: To score well in Asteroids, you don't just shoot randomly—you need to position yourself to avoid debris, prioritize larger asteroids first, and plan movements that set up multiple hits over time. Most baseline DRL methods are tuned to optimize for short-term rewards, making it hard to learn these longer, sequential decision-making patterns.
• Reward signal misalignment: While hitting an asteroid gives an immediate positive reward, the biggest risk is getting hit by shattered debris. The one-time negative reward for dying often isn't enough to teach the agent that a poorly timed shot (which splits an asteroid into smaller, faster pieces) increases its long-term chance of getting destroyed. This creates a noisy reward signal that doesn't properly link actions to their delayed consequences.
• High environmental variance: Asteroid spawn positions and movement paths have inherent randomness. DRL agents rely on consistent state transition patterns to learn effectively, and this high variance makes it tough for them to converge to a stable, reliable policy.

It's worth mentioning that some more advanced approaches—like those adding memory layers or using hierarchical RL—have shown marginal improvements, but even they rarely match human-level performance.

内容的提问来源于stack exchange，提问作者hipoglucido