月球着陆器强化学习训练Sequential对象不可下标访问报错解决

根本原因 #

你调用lander_play_one_step函数时传参顺序错误：

• 函数定义的参数顺序为 def lander_play_one_step(env,model, obs, loss_fn)
• 你在play_multiple_episodes中调用时写为 lander_play_one_step(env, obs, model, loss_fn)，把obs和model的位置颠倒了。

这就导致单步函数内的形参obs实际拿到的是Sequential模型对象，执行obs[np.newaxis]时本质是对模型做下标索引，因此触发'Sequential' object is not subscriptable报错。

其他伴随问题 #

除了传参顺序错误外，你的代码还存在几处逻辑错误需要同步修复：

1. 损失函数选择错误：LunarLander-v2是4分类动作任务，不能用二分类交叉熵，要替换为稀疏分类交叉熵sparse_categorical_crossentropy
2. 返回值不匹配：lander_play_one_step当前返回obs, reward, done , info，但调用时接收的第四个值是grads，需要调整返回值顺序
3. 列表推导式变量名错误：计算平均梯度时enumerate(final_reward)写错，应为enumerate(final_rewards)
4. 缺少依赖函数：你代码中调用的discount_and_normalize_rewards没有给出实现，需要补充该函数

修复代码 #

1. 调整调用参数顺序 #

修改play_multiple_episodes中调用单步函数的代码：

# 修正参数顺序：env, model, obs, loss_fn
obs, reward, done, grads = lander_play_one_step(env, model, obs, loss_fn)

2. 调整单步函数返回值 #

def lander_play_one_step(env,model, obs, loss_fn):
    with tf.GradientTape() as tape:
       probas = model(obs[np.newaxis])
       logits = tf.math.log(probas + keras.backend.epsilon())
       action = tf.random.categorical(logits, num_samples=1)
       loss = tf.reduce_mean(loss_fn(action, probas))

    grads = tape.gradient(loss, model.trainable_variables)
    obs, reward, done, info = env.step(int(action[0,0].numpy()))
    # 返回梯度而非info
    return obs, reward, done, grads

3. 替换损失函数 #

loss_fn = keras.losses.sparse_categorical_crossentropy

4. 修正梯度计算逻辑 #

mean_grads = tf.reduce_mean( [final_reward * all_grads[episode_index][step][var_index] 
                              for episode_index, final_rewards in enumerate(all_final_rewards)
                                    for step , final_reward in enumerate(final_rewards)], axis=0 )

5. 补充奖励折扣归一化函数 #

def discount_rewards(rewards, discount_factor):
    discounted = np.zeros(len(rewards))
    running_add = 0
    for t in reversed(range(len(rewards))):
        running_add = running_add * discount_factor + rewards[t]
        discounted[t] = running_add
    return discounted

def discount_and_normalize_rewards(all_rewards, discount_factor):
    all_discounted_rewards = [discount_rewards(rewards, discount_factor) for rewards in all_rewards]
    flat_rewards = np.concatenate(all_discounted_rewards)
    reward_mean = flat_rewards.mean()
    reward_std = flat_rewards.std()
    return [(discounted_rewards - reward_mean)/reward_std for discounted_rewards in all_discounted_rewards]

内容的提问来源于stack exchange，提问作者bfdd