使用StableBaselines3的PPO训练PettingZoo智能体遇报错求助

尝试用StableBaselines3的单智能体PPO算法，在PettingZoo的协作环境中做参数共享训练，但将环境传入PPO构造函数时出现以下报错：

The algorithm only supports (<class 'gym.spaces.box.Box'>, <class 'gym.spaces.discrete.Discrete'>, <class 'gym.spaces.multi_discrete.MultiDiscrete'>, <class 'gym.spaces.multi_binary.MultiBinary'>) as action spaces but Box(-1.0, 1.0, (1,), float32) was provided

完整复现代码 #

from pettingzoo.butterfly import pistonball_v6
from pettingzoo.utils.conversions import aec_to_parallel
import supersuit as ss

from stable_baselines3.ppo import CnnPolicy
from stable_baselines3 import PPO


def main():
    # Initialize environment
    env = pistonball_v6.env(n_pistons=20, 
                        time_penalty=-0.1, 
                        continuous=True, 
                        random_drop=True, 
                        random_rotate=True, 
                        ball_mass=0.75, 
                        ball_friction=0.3, 
                        ball_elasticity=1.5,
                        max_cycles=125)
    # Reduce the complexity of the observation by considering only the blue channel
    env = ss.color_reduction_v0(env, mode='B')
    # Resize the observation to reduce dimension
    env = ss.resize_v1(env, x_size=84, y_size=84)
    # In order to let the policy learn based on the ball's velocity and acceleration, 
    # we include the last 3 consecutive frames in the observation
    env = ss.frame_stack_v1(env,3)
    # This is for using stable baselines
    env = aec_to_parallel(env)
    env = ss.pettingzoo_env_to_vec_env_v1(env)
    
    # prepare the environment to use stablebaselines
    env = ss.concat_vec_envs_v1(env, 2, num_cpus=1, base_class='stable_baselines3')
    # PPO
    model = PPO(CnnPolicy, 
                env, 
                verbose=3,
                gamma=0.95,
                n_steps=256,
                ent_coef=0.0905168,
                learning_rate=0.00062211,
                vf_coef=0.042202,
                max_grad_norm=0.9,
                gae_lambda=0.99,
                n_epochs=5,
                clip_range=0.3,
                batch_size=256)

    model.learn(total_timesteps=100000)

    pass


if __name__ == "__main__":
    main()

解决方法 #

核心原因 #

报错看似矛盾，实际是因为当前PettingZoo默认使用gymnasium的Box空间类型，而旧版本StableBaselines3仅兼容原生gym的空间类型，两者属于不同类，导致类型检查失败。此外，多智能体环境转换时的配置也可能影响动作空间的正确传递。

修复步骤 #

1. 升级StableBaselines3到兼容gymnasium的版本
   执行以下命令升级：

   pip install --upgrade stable-baselines3[extra]
   

   最新版SB3已全面支持gymnasium的空间类型，可直接解决类型不匹配问题。

2. 优化多智能体环境转换（适配参数共享）
   修改环境转换代码，增加flatten=True参数，确保多智能体的Dict动作/观察空间被展平为单个数组，适配单智能体PPO的参数共享训练逻辑：

   # 原转换代码
   env = aec_to_parallel(env)
   env = ss.pettingzoo_env_to_vec_env_v1(env)
   # 修改为
   env = aec_to_parallel(env)
   env = ss.pettingzoo_env_to_vec_env_v1(env, flatten=True)
   

3. 确认策略与动作空间匹配
   CnnPolicy本身支持连续动作空间，代码中continuous=True的配置已确保环境使用连续Box动作空间，无需额外修改策略类型。

修复后完整代码 #

from pettingzoo.butterfly import pistonball_v6
from pettingzoo.utils.conversions import aec_to_parallel
import supersuit as ss

from stable_baselines3.ppo import CnnPolicy
from stable_baselines3 import PPO


def main():
    # Initialize environment
    env = pistonball_v6.env(n_pistons=20, 
                        time_penalty=-0.1, 
                        continuous=True, 
                        random_drop=True, 
                        random_rotate=True, 
                        ball_mass=0.75, 
                        ball_friction=0.3, 
                        ball_elasticity=1.5,
                        max_cycles=125)
    # Reduce observation complexity (blue channel only)
    env = ss.color_reduction_v0(env, mode='B')
    # Resize observation
    env = ss.resize_v1(env, x_size=84, y_size=84)
    # Stack last 3 frames to capture motion info
    env = ss.frame_stack_v1(env,3)
    # Convert to parallel environment
    env = aec_to_parallel(env)
    # Convert to vec_env for SB3, flatten spaces for parameter sharing
    env = ss.pettingzoo_env_to_vec_env_v1(env, flatten=True)
    # Concatenate vector environments
    env = ss.concat_vec_envs_v1(env, 2, num_cpus=1, base_class='stable_baselines3')
    # Initialize PPO model
    model = PPO(CnnPolicy, 
                env, 
                verbose=3,
                gamma=0.95,
                n_steps=256,
                ent_coef=0.0905168,
                learning_rate=0.00062211,
                vf_coef=0.042202,
                max_grad_norm=0.9,
                gae_lambda=0.99,
                n_epochs=5,
                clip_range=0.3,
                batch_size=256)

    model.learn(total_timesteps=100000)


if __name__ == "__main__":
    main()

内容的提问来源于stack exchange，提问作者Onil90