You need to enable JavaScript to run this app. PyTorch强化学习项目运行遇输入类型与张量dtype不匹配错误求助
优惠活动
大模型
产品
解决方案
定价
更多

PyTorch强化学习项目运行遇输入类型与张量dtype不匹配错误求助

强化学习开源项目运行异常排查记录

故障整体表现

  • 尝试运行获取到的强化学习开源项目代码时程序持续崩溃,先后出现TypeError: only size-1 arrays can be converted to Python scalars报错
  • 已花费两日时间排查,初步判断问题为张量dtype不兼容,尝试将张量类型修改为DoubleTensor或float64时又会触发其他错误,始终无法定位具体需要修改的模块及对应修改方法
  • 已收到反馈指出问题根源为传入的state参数类型不符合PyTorch数值类型要求,但尝试将state转换为对应浮点类型时,会触发列表无法直接转换为float32的报错,持续排查未能定位根因
  • 参考的开源项目为股票交易方向的强化学习实现

涉及的核心代码片段

Policy网络模型定义

class Policy(nn.Module):
    def __init__(self):      
        super(Policy, self).__init__()
        self.input_layer = nn.Linear(11, 128)
        self.hidden_1 = nn.Linear(128, 128)
        self.hidden_2 = nn.Linear(32,31)
        self.hidden_state = torch.tensor(torch.zeros(2,1,32)).cuda()
        self.rnn = nn.GRU(128, 32, 2)
        self.action_head = nn.Linear(31, 5)
        self.value_head = nn.Linear(31, 1)
        self.saved_actions = []
        self.rewards = []

    def reset_hidden(self):
        self.hidden_state = torch.tensor(torch.zeros(2,1,32)).cuda()
        
    def forward(self, x): 
        x = torch.tensor(x).cuda()
        x = torch.sigmoid(self.input_layer(x))
        x = torch.tanh(self.hidden_1(x))
        x, self.hidden_state = self.rnn(x.view(1,-1,128), self.hidden_state.data)
        x = F.relu(self.hidden_2(x.squeeze()))
        action_scores = self.action_head(x)
        state_values = self.value_head(x)
        return F.softmax(action_scores, dim=-1), state_values

        def forward(self, x):
          conv_out = self.conv(x).view(x.size()[0], -1)
          val = self.fc_val(conv_out)
          adv = self.fc_adv(conv_out)
          return val + (adv - adv.mean(dim=1, keepdim=True))
    
    def act(self, state):
        probs, state_value = self.forward(state)
        m = Categorical(probs)
        action = m.sample()
        if action == 1 and env.state[0] < 1: action = torch.LongTensor([2]).squeeze().cuda.DoubleTensor()
        if action == 4 and env.state[1] < 1: action = torch.LongTensor([2]).squeeze().cuda.DoubleTensor()
        if action == 6 and env.state[2] < 1: action = torch.LongTensor([2]).squeeze().cuda.DoubleTensor()
        self.saved_actions.append((m.log_prob(action), state_value))
        return action.item()

程序运行逻辑

env.reset()
# In case you're running this a second time with the same model, delete the gradients
del model.rewards[:]
del model.saved_actions[:]

gamma = 0.9
log_interval = 60

def finish_episode():
    R = 0
    saved_actions = model.saved_actions
    policy_losses = []
    value_losses = []
    rewards = []
    for r in model.rewards[::-1]:
        R = r + (gamma * R)
        rewards.insert(0, R)
    rewards = torch.tensor(rewards)
    
    epsilon = (torch.rand(1) / 1e4) - 5e-5
    # With different architectures, I found the following standardization step sometimes
    # helpful, sometimes unhelpful.
    #
    rewards = (rewards - rewards.mean()) / (rewards.std(unbiased=False) + epsilon)
    # Alternatively, comment it out and use the following line instead:
    rewards += epsilon
    
    for (log_prob, value), r in zip(saved_actions, rewards):
        reward = torch.tensor(r - value.item()).cuda()
        policy_losses.append(-log_prob * reward)
        value_losses.append(F.smooth_l1_loss(value, torch.tensor([r]).cuda()))
        
    optimizer.zero_grad()
    loss = torch.stack(policy_losses).sum() + torch.stack(value_losses).sum()
    loss = torch.clamp(loss, -1e-5, 1e5)
    loss.backward()
    optimizer.step()
    del model.rewards[:]
    del model.saved_actions[:]

running_reward = 0
for episode in range(0, 4000):
    state = env.reset()
    state = state.type(torch.float32)
    reward = 0
    done = False
    msg = None 
    while not done:
        action = model.act(state)
        state, reward, done, msg = env.step(action)
        model.rewards.append(reward)
        if done:
            break
    running_reward = running_reward * (1 - 1/log_interval) + reward * (1/log_interval)
    finish_episode()
    # Resetting the hidden state seems unnecessary - it's effectively random from the previous
    # episode anyway, more random than a bunch of zeros.
    # model.reset_hidden()
    if msg["msg"] == "done" and env.portfolio_value() > env.starting_portfolio_value * 1.1 and running_reward > 500:
        print("Early Stopping: " + str(int(reward)))
        break
    if episode % log_interval == 0:
        print("""Episode {}: started at {:.1f}, finished at {:.1f} because {} @ t={}, \
last reward {:.1f}, running reward {:.1f}""".format(episode, env.starting_portfolio_value, \
              env.portfolio_value(), msg["msg"], env.cur_timestep, reward, running_reward))

运行捕获的错误日志

/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:18: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-74-21b617d2e36f> in <module>()
     46     msg = None
     47     while not done:
---> 48         action = model.act(state)
     49         state, reward, done, msg = env.step(action)
     50         model.rewards.append(reward)

4 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1751     if has_torch_function_variadic(input, weight):
   1752         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
-> 1753     return torch._C._nn.linear(input, weight, bias)
   1754 
   1755 

RuntimeError: expected scalar type Double but found Float

报错触发链路明确为:Episode循环调用model.act(state)方法时,在PyTorch线性层计算环节出现dtype不匹配,线性层权重为Double类型、输入张量为Float类型,二者无法参与计算。


内容的提问来源于stack exchange,提问作者Joanna Kastelik

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.10.27 15:54:03