在OpenAI Gym MountainCar-v0实现Q-learning遇ValueError报错求助

报错原因 #

你使用的是新版本OpenAI Gym（0.26+），这个版本的API有两处关键更新导致报错：

• env.reset()不再仅返回状态数组，而是返回**(状态数组, 信息字典)**的元组，你直接把这个元组传给get_discrete_state函数，导致函数内的state是包含数组和字典的复合结构，而非预期的二维状态数组，做(state - env.observation_space.low)这类数组运算时，出现形状不兼容的错误。
• env.step(action)的返回值结构也有变化，原代码的取值逻辑会导致后续判断episode是否结束的逻辑不准确。

修复步骤 #

1. 修正env.reset()的调用 #

只取env.reset()返回值的第一个元素（状态数组）：

discrete_state = get_discrete_state(env.reset()[0])

2. 修正env.step(action)的返回值处理 #

新版本env.step(action)返回(new_state, reward, terminated, truncated, info)，其中terminated代表任务完成（比如到达目标），truncated代表超时等强制终止，两者任意一个为True都意味着episode结束，调整逻辑如下：

new_state, reward, terminated, truncated, info = env.step(action)
done = terminated or truncated

修复后的完整代码 #

import gym
import numpy as np
env = gym.make("MountainCar-v0", render_mode='human')

LEARNING_RATE = 0.1
DISCOUNT = 0.95
EPISODES = 25000
SHOW_EVERY = 2000
DISCRETE_OS_SIZE = [20] * len(env.observation_space.high)
discrete_os_win_size = (env.observation_space.high - env.observation_space.low) / np.array(DISCRETE_OS_SIZE)
q_table = np.random.uniform(low=-2, high=0, size=(DISCRETE_OS_SIZE + [env.action_space.n]))

def get_discrete_state(state):
    discrete_state = (state - env.observation_space.low) / discrete_os_win_size
    return tuple(discrete_state.astype(np.int))

for episode in range(EPISODES):
    if episode % SHOW_EVERY == 0:
        print(episode)
        render = True
    else:
        render = False

    discrete_state = get_discrete_state(env.reset()[0])

    done = False
    while not done:
        action = np.argmax(q_table[discrete_state])

        new_state, reward, terminated, truncated, info = env.step(action)
        done = terminated or truncated

        new_discrete_state = get_discrete_state(new_state)

        if render:
            env.render()

        if not done:
            max_future_q = np.max(q_table[new_discrete_state])
            current_q = q_table[discrete_state][action]
            new_q = (1 - LEARNING_RATE) * current_q + LEARNING_RATE * (reward + DISCOUNT * max_future_q)
            q_table[discrete_state][action] = new_q
        elif new_state[0] >= env.goal_position:
            q_table[discrete_state][action] = 0 
        discrete_state = new_discrete_state

env.close()

内容的提问来源于stack exchange，提问作者Julian_Morrow