使用交叉熵算法训练CartPole时遇ValueError问题求助

问题背景 #

正在学习强化学习，参考书籍官方代码用交叉熵算法训练CartPole游戏，运行时出现错误。

错误信息 #

UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ..\torch\csrc\utils\tensor_new.cpp:248.)
obs_t = torch.FloatTensor([obs])

Traceback (most recent call last):
File "C:\Users\tiklu\OneDrive\Desktop\DRL Resources\codes\basic_rl\cartpole_cross_entropy.py", line 101, in
for iter_no, batch in enumerate(iterate_batches(env, net, BATCH_SIZE)):
File "C:\Users\tiklu\OneDrive\Desktop\DRL Resources\codes\basic_rl\cartpole_cross_entropy.py", line 43, in iterate_batches
obs_t = torch.FloatTensor([obs])
ValueError: expected sequence of length 4 at dim 2 (got 0)

报错相关函数代码 #

def iterate_batches(env,net,batch_size):
    batch = [] #List of episodes
    episode_reward = 0.0
    episode_steps = []
    obs = env.reset()
    sm = nn.Softmax(dim=1)

    while True:
        #this line 
        obs_t = torch.FloatTensor([obs]) 
        # obs_t = obs_t.view(1, -1)  
        action_props_t = sm(net(obs_t)) #net expects a batch of items
        action_props = action_props_t.data.numpy()[0]

        action = np.random.choice(len(action_props),p = action_props)
        next_obs, reward, is_done, _, _ = env.step(action)

        episode_reward += reward
        episode_steps.append(EpisodeStep(observation=obs,action=action))

        if is_done:
            batch.append(Episode(reward=episode_reward,steps=episode_steps))
            episode_steps = []
            episode_reward = 0.0
            next_obs = env.reset()
            if len(batch)==batch_size:
                yield batch
                batch = []
        obs = next_obs

依赖版本 #

gym==0.26.2
numpy==1.24.3
tensorboardX==2.6
torch==2.0.0

解决方案 #

问题根源 #

1. Gym版本兼容性：gym 0.26.2中env.reset()返回(observation, info)元组，而书籍代码依赖的旧版本仅返回观测数组。当前obs是包含观测和信息的元组，而非纯观测数组，导致转Tensor时维度错误。
2. Tensor创建低效：直接从numpy数组列表创建Tensor的方式不符合PyTorch优化建议，触发性能警告。

修复步骤 #

1. 修正env.reset()的返回值处理：
   将所有env.reset()的赋值改为接收两个返回值：

   obs, _ = env.reset()
   

   next_obs, _ = env.reset()
   

2. 优化Tensor创建方式：
   替换obs_t = torch.FloatTensor([obs])为更高效且维度正确的写法：

   obs_t = torch.tensor(obs, dtype=torch.float32).unsqueeze(0)
   

   或通过numpy转换后再转Tensor：

   obs_t = torch.FloatTensor(np.array([obs]))
   

3. 维度兼容处理：
   若仍有维度不匹配问题，取消obs_t = obs_t.view(1, -1)的注释，确保输入网络的张量形状为(batch_size, obs_dim)。

修复后的完整函数代码 #

def iterate_batches(env,net,batch_size):
    batch = [] #List of episodes
    episode_reward = 0.0
    episode_steps = []
    obs, _ = env.reset()  # 修正reset返回值处理
    sm = nn.Softmax(dim=1)

    while True:
        # 优化Tensor创建并确保维度正确
        obs_t = torch.tensor(obs, dtype=torch.float32).unsqueeze(0)
        # 可选：obs_t = torch.FloatTensor(np.array([obs]))
        # obs_t = obs_t.view(1, -1)  # 维度异常时启用
        
        action_props_t = sm(net(obs_t))
        action_props = action_props_t.data.numpy()[0]

        action = np.random.choice(len(action_props),p = action_props)
        next_obs, reward, is_done, _, _ = env.step(action)

        episode_reward += reward
        episode_steps.append(EpisodeStep(observation=obs,action=action))

        if is_done:
            batch.append(Episode(reward=episode_reward,steps=episode_steps))
            episode_steps = []
            episode_reward = 0.0
            next_obs, _ = env.reset()  # 修正reset返回值处理
            if len(batch)==batch_size:
                yield batch
                batch = []
        obs = next_obs

内容的提问来源于stack exchange，提问作者Pulkit Prajapat