蒙特卡洛方法求解21点：Q值表异常问题排查

我正遵循Sutton和Barto的方法，实现蒙特卡洛（Monte Carlo）算法求解21点（Blackjack）问题。代码逻辑看似无误，但生成的Q值表显示在过多场景下建议选择stick（停牌）操作，无法确定是方法思路存在偏差还是代码实现有误。

智能体类代码（Epsilon-Greedy策略） #

class MonteCarloAgentEpsilonGreedy:
    def __init__(
        self,
        env,
        discount_factor,
        epsilon):

        self.env = env
        self.gamma = discount_factor
        self.epsilon = epsilon  # 探索率
        # 创建存储Q值的字典
        self.Q_values = defaultdict(lambda: np.zeros(env.action_space.n))
        self.Returns = defaultdict(lambda: np.zeros(env.action_space.n))
        self.N = defaultdict(lambda: np.zeros(env.action_space.n))

    def get_action(self, obs):
        """
        根据给定观测值选择动作（Epsilon-Greedy策略）

        参数:
            obs: 用于确定动作的观测值

        返回:
            action: 以1-epsilon概率选择Q值最高的动作，否则随机选择动作
        """

        if np.random.rand() < self.epsilon:
            action = self.env.action_space.sample() # 随机选择动作
        else:
            action = int(np.argmax(self.Q_values[obs])) # 选择Q值最高的动作

        return action

    def update_Q_values(self, episode):
        """
        根据回合数据更新Q值

        参数:
            episode: 包含(state, action, reward)元组的列表
        """
        G = 0
        for state, action, reward in reversed(episode):
            G = self.gamma * G + reward
            self.Returns[state][action] += G
            self.N[state][action] += 1
            # Q值更新规则
            self.Q_values[state][action] = self.Returns[state][action] / self.N[state][action]

主函数代码 #

if __name__ == "__main__":
    env = gym.make('Blackjack-v1', natural=False, sab=False)

    # 创建MonteCarloAgent实例
    agent = MonteCarloAgentEpsilonGreedy(
        env, discount_factor=0.9, epsilon=0.1)

    num_episodes = 1000000

    for e in range(num_episodes):

        episode = []
        terminated = False
        truncated = False
        # 随机初始化状态
        observation, info = env.reset()

        while (not terminated and not truncated):  # 回合循环

            action = agent.get_action(observation)

            next_obs, reward, terminated, truncated, info = env.step(action)
            episode.append((observation, action, reward))
        agent.update_Q_values(episode)
    

    env.close()

部分观测对应的Q值示例 #

• (15, 10, 0): array([-0.57322077, -0.57813051])
• (19, 3, 0): array([ 0.39937642, -0.67754011])
• (17, 10, 0): array([-0.45902484, -0.68447894])
• (11, 8, 0): array([-0.47658631, -0.47728385])
• (12, 10, 0): array([-0.54324405, -0.5438698 ])
• (20, 10, 0): array([ 0.44418773, -0.84017038])
• (11, 10, 0): array([-0.54170763, -0.54247852])
• (15, 3, 0): array([-0.24095023, -0.49996364])
• (18, 6, 0): array([ 0.28397257, -0.6047619 ])
• (20, 4, 0): array([ 0.65904186, -0.87462687])
• (13, 8, 0): array([-0.50007986, -0.50656757])
• (13, 6, 0): array([-0.14338235, -0.38048843])
• (17, 5, 0): array([-0.03217932, -0.57848101])

Q值表可视化结果 #

注：蓝色代表stick（停牌）场景，红色代表hit（要牌）场景

内容的提问来源于stack exchange，提问作者Mesko