使用SARSA算法实现Blackjack-v1环境时遇解包错误求排查

问题背景 #

尝试用SARSA算法实现OpenAI Gym的Blackjack-v1环境，核心代码如下：

import numpy as np
import gym

# SARSA parameters
alpha = 0.1
gamma = 0.99
epsilon = 0.1

# Function to discretize state space
def discretize_state(state):
    if isinstance(state, tuple):
        # If the state is a tuple, assume it has three elements
        player_sum, dealer_card, usable_ace = state
    else:
        # If the state is not a tuple, assume it has two elements and no usable_ace
        player_sum, dealer_card = state
        usable_ace = False

    player_sum = min(21, player_sum // 10)
    dealer_card = min(10, dealer_card - 1)
    return (player_sum, dealer_card, int(usable_ace))

# Function to choose an action using epsilon-greedy policy
def choose_action(Q, state):
    if np.random.uniform(0, 1) < epsilon:
        return np.random.choice([0, 1])  # 0 for stick, 1 for hit
    else:
        return np.argmax(Q[state])

# Function to update Q-values using SARSA
def update_Q(Q, state, action, reward, next_state, next_action):
    Q[state][action] += alpha * (reward + gamma * Q[next_state][next_action] - Q[state][action])

# Function to train the agent using SARSA
def train_agent(env, num_episodes):
    Q = np.zeros((10, 10, 2, 2))  # State space discretization

    for episode in range(num_episodes):
        state = discretize_state(env.reset())
        action = choose_action(Q, state)
        done = False

        while not done:
            next_state, reward, done, _ = env.step(action)
            next_state = discretize_state(next_state)
            next_action = choose_action(Q, next_state)
            update_Q(Q, state, action, reward, next_state, next_action)
            state, action = next_state, next_action

    return Q

# Function to evaluate the trained agent
def evaluate_agent(Q, env, num_episodes):
    total_reward = 0
    for _ in range(num_episodes):
        state = discretize_state(env.reset())
        action = choose_action(Q, state)
        done = False

        while not done:
            next_state, reward, done, _ = env.step(action)
            next_state = discretize_state(next_state)
            next_action = choose_action(Q, next_state)
            total_reward += reward
            state, action = next_state, next_action

    return total_reward / num_episodes

if __name__ == "__main__":
    # Create Blackjack environment
    env = gym.make('Blackjack-v1')

    # Train the agent
    num_episodes = 100000
    Q_values = train_agent(env, num_episodes)

    # Evaluate the trained agent
    num_eval_episodes = 1000
    avg_reward = evaluate_agent(Q_values, env, num_eval_episodes)
    print(f"Average reward over {num_eval_episodes} episodes: {avg_reward}")

运行后触发以下错误：

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_40536\2433186481.py in <module>
     73     # Train the agent
     74     num_episodes = 100000
---> 75     Q_values = train_agent(env, num_episodes)
     76 
     77     # Evaluate the trained agent

~\AppData\Local\Temp\ipykernel_40536\2433186481.py in train_agent(env, num_episodes)
     37 
     38     for episode in range(num_episodes):
---> 39         state = discretize_state(env.reset())
     40         action = choose_action(Q, state)
     41         done = False

~\AppData\Local\Temp\ipykernel_40536\2433186481.py in discretize_state(state)
     11     if isinstance(state, tuple):
     12         # If the state is a tuple, assume it has three elements
---> 13         player_sum, dealer_card, usable_ace = state
     14     else:
     15         # If the state is not a tuple, assume it has two elements and no usable_ace

ValueError: not enough values to unpack (expected 3, got 2)

错误原因 #

在Blackjack-v1环境中，env.reset()返回的初始状态是三元组（玩家手牌和、庄家明牌、是否有可用A），但当回合结束（done=True）时，env.step(action)返回的next_state会变成二元组（仅玩家和庄家的最终手牌和）。你的discretize_state函数默认所有元组都是三元组，导致解包时触发错误。

解决方案 #

修改discretize_state函数，通过元组长度判断状态类型后再解包，同时修正离散化逻辑的冗余问题：

def discretize_state(state):
    if isinstance(state, tuple):
        # 根据元组长度区分正常状态和终端状态
        if len(state) == 3:
            player_sum, dealer_card, usable_ace = state
        elif len(state) == 2:
            # 终端状态无可用A标识
            player_sum, dealer_card = state
            usable_ace = False
        else:
            raise ValueError(f"Unexpected state tuple length: {len(state)}")
    else:
        player_sum, dealer_card = state
        usable_ace = False

    # 合理离散化：玩家和分为0-9、10-19、20-21三个区间
    player_sum = min(2, player_sum // 10)
    # 庄家牌转为0-9索引（原牌面1-10）
    dealer_card = min(9, dealer_card - 1)
    return (player_sum, dealer_card, int(usable_ace))

额外优化说明 #

• 原代码中player_sum = min(21, player_sum // 10)逻辑冗余：player_sum//10的结果仅为0、1、2，用min(21, ...)完全没必要，修正为min(2, player_sum//10)更贴合离散化需求。
• 若想节省Q数组空间，可将Q = np.zeros((10, 10, 2, 2))改为Q = np.zeros((3, 10, 2, 2))，因为玩家和的离散结果只有3种可能。

其他注意点 #

• 当done=True时，终端状态的next_action不会影响后续更新（回合循环直接终止），当前SARSA逻辑可正常运行。
• 测试时可先将训练回合数改为1000，快速验证错误是否修复。

内容的提问来源于stack exchange，提问作者Detox34