构建MasterMind游戏RL环境遇MultiDiscrete相关报错求助

问题背景 #

我正在为MasterMind游戏构建强化学习算法，采用MultiDiscrete动作空间（4个槽位，每个槽位含6种颜色）与MultiDiscrete观测空间（维度为2×4），并自定义了Gym环境，开发中先后遇到两个错误：

初始报错：使用MultiInputActorCriticPolicy触发AttributeError #

相关代码

import gym as gym
from gym import Env
from gym.spaces import Discrete, Box, MultiDiscrete, Dict
from stable_baselines3.common.policies import MultiInputActorCriticPolicy
import numpy as np

action_space = MultiDiscrete(np.array([6,6,6,6]), dtype=int)
observation_space = MultiDiscrete(np.array([4,4]), dtype=int)

class MasterMindEnv(Env):
    def __init__(self) -> None:
        super(MasterMindEnv, self).__init__()
        self.action_space = action_space
        self.observation_space = observation_space

    def step(self, action:np.ndarray):
        # pass_action(action)
        # output = get_output()
        # print(output)

        reward = 1  # 示例值，原代码中是output[0]+output[1]
        print(reward)
        
        done = False
        info = {}

        return observation_space.sample(), reward, done, info

    def reset(self):
        return self.observation_space.sample()

env = MasterMindEnv()
model = A2C(MultiInputActorCriticPolicy, env)
model.learn(total_timesteps=1000)

报错信息

AttributeError                            Traceback (most recent call last)
c:\...\model.ipynb 单元格10 in <module>
----> 1 model = A2C(MultiInputActorCriticPolicy, env)
      2 model.learn(total_timesteps=1000)

...
--> 258 for key, subspace in observation_space.spaces.items():
    259     if is_image_space(subspace):
    260         extractors[key] = NatureCNN(subspace, features_dim=cnn_output_dim)

AttributeError: 'MultiDiscrete'对象没有属性'spaces'

解决方法

MultiInputActorCriticPolicy是专门为Dict/Tuple类型的多输入观测空间设计的，你的观测空间是单一的MultiDiscrete，不属于多输入空间，因此直接使用MlpPolicy即可。

更新后报错：使用MlpPolicy触发RuntimeError #

更新后代码

import gym as gym
from gym import Env
from gym.spaces import MultiDiscrete
import numpy as np
from stable_baselines3 import A2C

class MasterMindEnv(Env):
    def __init__(self) -> None:
        super(MasterMindEnv, self).__init__()
        self.action_space = MultiDiscrete(np.array([6,6,6,6]), dtype=int)
        self.observation_space = MultiDiscrete(np.array([4,4]), dtype=int)

    def step(self, action:np.ndarray):
        output = self.observation_space.sample()
        reward = output[0] + output[1]
        done = False
        if (reward == 8):
            done = True
        info = {}
        return output.astype(np.int64), float(reward), done, info

    def reset(self):
        return self.observation_space.sample().astype(np.int64)

env = MasterMindEnv()

model = A2C("MlpPolicy", env)
model.learn(total_timesteps=1000)

报错信息

RuntimeError                              Traceback (most recent call last)
c:\...\model.ipynb 单元格9 in <module>
      1 model = A2C("MlpPolicy", env)
----> 2 model.learn(total_timesteps=1000)

...
--> 464     return th.as_tensor(obs).to(device)
    465 elif isinstance(obs, dict):
    466     return {key: th.as_tensor(_obs).to(device) for (key, _obs) in obs.items()}

RuntimeError: 无法推断numpy.int32的dtype

解决方法

1. 修正变量引用错误：step方法中原本误用全局的observation_space，改为调用实例的self.observation_space，避免作用域问题。
2. 统一观测的numpy dtype：Stable Baselines3的PyTorch后端对int32存在兼容性问题，将返回的观测转换为int64类型（通过.astype(np.int64)），确保PyTorch可以正确推断数据类型。

内容的提问来源于stack exchange，提问作者AR_Jini