SB3组合vec_env与AtariWrapper报AttributeError问题求助

在尝试将Stable Baselines 3（SB3）的向量环境（vec_env）与AtariWrapper结合使用时，触发如下错误：

AttributeError: 'DummyVecEnv' object has no attribute 'get_action_meanings'

报错代码 #

import gymnasium as gym

from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.atari_wrappers import AtariWrapper

# Parallel environments
# vec_env = gym.make("PongNoFrameskip-v4")
vec_env = make_vec_env("PongNoFrameskip-v4", n_envs=2, seed=3)
vec_env = AtariWrapper(vec_env)
model = PPO("CnnPolicy", vec_env, verbose=1, n_steps=128, n_epochs=4,
            batch_size=256, learning_rate=2.5e-4, clip_range=0.1,
            vf_coef=0.5, ent_coef=0.01)
model.learn(total_timesteps=1e7)
model.save("ppo_cartpole")

完整报错信息 #

A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Traceback (most recent call last):
  File "D:\q_learning\sb3_ppo.py", line 10, in <module>
    vec_env = AtariWrapper(vec_env)
  File "C:\Users\thoma\anaconda3\envs\torch_2\lib\site-packages\stable_baselines3\common\atari_wrappers.py", line 294, in __init__
    env = NoopResetEnv(env, noop_max=noop_max)
  File "C:\Users\thoma\anaconda3\envs\torch_2\lib\site-packages\stable_baselines3\common\atari_wrappers.py", line 57, in __init__
    assert env.unwrapped.get_action_meanings()[0] == "NOOP"  # type: ignore[attr-defined]
AttributeError: 'DummyVecEnv' object has no attribute 'get_action_meanings'

Process finished with exit code 1

可正常运行的代码（未使用向量环境） #

vec_env = gym.make("PongNoFrameskip-v4")
# vec_env = make_vec_env("PongNoFrameskip-v4", n_envs=2, seed=3)
vec_env = AtariWrapper(vec_env)
model = PPO("CnnPolicy", vec_env, verbose=1, n_steps=128, n_epochs=4,
            batch_size=256, learning_rate=2.5e-4, clip_range=0.1,
            vf_coef=0.5, ent_coef=0.01)
model.learn(total_timesteps=1e7)
model.save("ppo_cartpole")

AtariWrapper是为单个Atari环境设计的，初始化时会调用环境的get_action_meanings()方法验证NOOP动作存在。但make_vec_env创建的DummyVecEnv是向量环境容器，本身没有这个方法——真正的Atari环境是向量环境内部的子环境，直接给向量环境套AtariWrapper自然会报错。

正确的做法是给每个子环境单独应用AtariWrapper，而非给整个向量环境加Wrapper。有两种常用实现方式：

方式一：使用make_vec_env的wrapper_class参数 #

直接通过wrapper_class指定要给每个子环境应用的Wrapper：

import gymnasium as gym

from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.atari_wrappers import AtariWrapper

# 给每个子环境自动应用AtariWrapper
vec_env = make_vec_env("PongNoFrameskip-v4", n_envs=2, seed=3, wrapper_class=AtariWrapper)
model = PPO("CnnPolicy", vec_env, verbose=1, n_steps=128, n_epochs=4,
            batch_size=256, learning_rate=2.5e-4, clip_range=0.1,
            vf_coef=0.5, ent_coef=0.01)
model.learn(total_timesteps=1e7)
model.save("ppo_pong")

方式二：自定义环境生成函数 #

如果需要组合多个Wrapper，可以自定义生成环境的函数，再传给make_vec_env：

import gymnasium as gym

from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.atari_wrappers import AtariWrapper

def make_custom_atari_env(env_id, seed):
    env = gym.make(env_id)
    # 这里可以添加多个Wrapper，比如先加AtariWrapper再自定义其他Wrapper
    env = AtariWrapper(env)
    env.seed(seed)
    return env

vec_env = make_vec_env(make_custom_atari_env, env_id="PongNoFrameskip-v4", n_envs=2, seed=3)
model = PPO("CnnPolicy", vec_env, verbose=1, n_steps=128, n_epochs=4,
            batch_size=256, learning_rate=2.5e-4, clip_range=0.1,
            vf_coef=0.5, ent_coef=0.01)
model.learn(total_timesteps=1e7)
model.save("ppo_pong")

内容的提问来源于stack exchange，提问作者desert_ranger