OpenAI Gym MountainCar项目Q学习代码出现IndexError问题求助

问题场景 #

开发OpenAI Gym的MountainCar-v0项目时，运行Q学习代码遇到以下错误：

IndexError: arrays used as indices must be of integer (or boolean) type

已用print(type(state))确认初始state是整数，但问题仍未解决。完整代码及错误信息如下：

原代码 #

import gym
import numpy as np
import pickle as pkl

mountEnv = gym.make("MountainCar-v0", render_mode = "human")

q_table = np.zeros(shape = (48,4))
# Parameters
EPSILON = 0.1
ALPHA = 0.9
GAMMA = 0.9
NUM_EPISODES = 500

def policy(state, explore=0.0):
    action = int(np.argmax(q_table[state]))
    if np.random.random() <= explore:
        action = int(np.random.randint(0,2))
    return action

for episode in range(NUM_EPISODES):

    done = False
    total_reward = 0
    episode_length = 0

    mountEnv.reset()
    state = 0
    while not done:
        action = policy(state, EPSILON)
        next_state, reward, done, _, _ = mountEnv.step(action)
        next_action = policy(next_state)

        q_table[state][action] += ALPHA + (reward + GAMMA * q_table[next_state][next_action] - q_table[state][action])

        state = next_state

        total_reward += reward
        episode_length += 1
    print("Episode:", episode, "Length:", episode_length, "Total Reward: ", total_reward)

mountEnv.close()
pkl.dump(q_table, open("q_learning_q_table.pkl", "wb"))

完整错误信息 #

/usr/local/lib/python3.10/dist-packages/gym/utils/passive_env_checker.py:233: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated NumPy 1.24)
  if not isinstance(terminated, (bool, np.bool8)):
Traceback (most recent call last):
  File "/home/matteo/Downloads/Q-Learning-and-SARSA-Mountain-Car-v0-main/matteos-code.py", line 31, in <module>
    next_action = policy(next_state)
  File "/home/matteo/Downloads/Q-Learning-and-SARSA-Mountain-Car-v0-main/matteos-code.py", line 15, in policy
    action = int(np.argmax(q_table[state]))
IndexError: arrays used as indices must be of integer (or boolean) type

Process finished with exit code 1

错误原因分析 #

1. State类型错误：MountainCar-v0的环境返回的state是连续二维数组（包含小车位置和速度两个浮点数），不是整数。你手动将初始state设为0，但step()返回的next_state是数组，直接用它索引q_table必然报错。
2. Q表维度错误：MountainCar-v0的动作空间是3个动作（向左、不动、向右），但你定义的q_table是(48,4)，多了一个无效动作维度。
3. 动作范围错误：epsilon贪心策略中，randint(0,2)只能生成0、1两个动作，漏掉了第三个动作。
4. Q值更新公式错误：正确的Q学习更新公式是Q(s,a) += α * (r + γ * maxQ(s',a') - Q(s,a))，你写成了ALPHA + (...)，把乘法写成了加法。
5. Reset处理错误：mountEnv.reset()会返回初始状态，你没有接收这个值，而是手动设为0，不符合环境逻辑。

修复方案 #

1. 离散化连续状态 #

由于环境的state是连续值，需要将其映射为整数索引才能用于Q表查找。可以通过线性映射将位置和速度分别离散化，再组合成一个整数状态值：

• 位置范围：[-1.2, 0.6]，分成24个区间
• 速度范围：[-0.07, 0.07]，分成24个区间
• 总状态数：24×24=576

2. 修正Q表及参数 #

• Q表维度改为(576, 3)，对应所有离散状态和3个动作
• 修正动作生成范围为randint(0,3)
• 修正Q值更新公式中的乘法逻辑

3. 正确处理环境状态 #

接收reset()返回的初始状态，将其离散化后作为初始state；每次step()返回的next_state也需要先离散化再使用。

修复后完整代码 #

import gym
import numpy as np
import pickle as pkl

mountEnv = gym.make("MountainCar-v0", render_mode="human")

# 离散化参数：位置分24段，速度分24段，总状态数24*24=576
POS_BINS = 24
VEL_BINS = 24
ACTION_SPACE = 3  # MountainCar-v0有3个动作：0=左，1=不动，2=右

# 初始化Q表
q_table = np.zeros(shape=(POS_BINS * VEL_BINS, ACTION_SPACE))

# 强化学习参数
EPSILON = 0.1
ALPHA = 0.1  # 原学习率0.9过高，调整为更合理的值
GAMMA = 0.9
NUM_EPISODES = 500

def discretize_state(state):
    """将连续的状态（位置、速度）转换为离散整数索引"""
    pos, vel = state
    # 计算位置的离散索引
    pos_min = mountEnv.observation_space.low[0]
    pos_max = mountEnv.observation_space.high[0]
    pos_idx = int(np.floor((pos - pos_min) / (pos_max - pos_min) * POS_BINS))
    pos_idx = np.clip(pos_idx, 0, POS_BINS - 1)  # 防止越界
    
    # 计算速度的离散索引
    vel_min = mountEnv.observation_space.low[1]
    vel_max = mountEnv.observation_space.high[1]
    vel_idx = int(np.floor((vel - vel_min) / (vel_max - vel_min) * VEL_BINS))
    vel_idx = np.clip(vel_idx, 0, VEL_BINS - 1)
    
    # 组合成唯一的状态索引
    return pos_idx * VEL_BINS + vel_idx

def policy(state_idx, explore=0.0):
    """ε-贪心策略"""
    if np.random.random() <= explore:
        # 随机选择动作
        return np.random.randint(0, ACTION_SPACE)
    else:
        # 选择Q值最大的动作
        return np.argmax(q_table[state_idx])

for episode in range(NUM_EPISODES):
    done = False
    total_reward = 0
    episode_length = 0
    
    # 获取初始状态并离散化
    initial_state, _ = mountEnv.reset()
    state_idx = discretize_state(initial_state)
    
    while not done:
        action = policy(state_idx, EPSILON)
        next_state, reward, done, _, _ = mountEnv.step(action)
        next_state_idx = discretize_state(next_state)
        
        # Q学习更新公式
        current_q = q_table[state_idx][action]
        max_next_q = np.max(q_table[next_state_idx])
        q_table[state_idx][action] += ALPHA * (reward + GAMMA * max_next_q - current_q)
        
        state_idx = next_state_idx
        total_reward += reward
        episode_length += 1
    
    print(f"Episode: {episode} | Length: {episode_length} | Total Reward: {total_reward:.2f}")

mountEnv.close()
pkl.dump(q_table, open("q_learning_q_table.pkl", "wb"))

额外说明 #

• 原代码中学习率ALPHA设为0.9过高，容易导致Q值震荡，调整为0.1更合适
• 离散化的区间数量可以根据需求调整，区间越多Q表越精准，但训练速度会变慢

内容的提问来源于stack exchange，提问作者user21955947