keras-rl2的DQN Agent在Taxi-v3训练时遇ValueError求助

问题背景 #

尝试用keras-rl2的DQNAgent解决OpenAI Gym的Taxi-v3问题，操作流程如下：

• 从gym打开Taxi-v3环境
• 用Keras Sequential API构建含Embedding和Dense层的深度学习模型
• 导入Epsilon Greedy策略与Sequential Memory队列
• 组装DQNAgent并编译

拟合模型时触发错误，相同代码在CartPole环境下正常运行，推测是Taxi-v3状态为标量（共500种），而CartPole是4元素数组导致。同时希望将单轮训练步数扩展至200以上（已设置env._max_episode_steps=5000）。

错误回溯 #

Training for 1000000 steps ...
Interval 1 (0 steps performed)

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-180-908ee27d8389> in <module>
      1 agent.compile(Adam(lr=0.001),metrics=['mae'])
----> 2 agent.fit(env, nb_steps=1000000, visualize=False, verbose=1, nb_max_episode_steps=99, log_interval=100000)

/usr/local/lib/python3.8/dist-packages/rl/core.py in fit(self, env, nb_steps, action_repetition, callbacks, verbose, visualize, nb_max_start_steps, start_step_policy, log_interval, nb_max_episode_steps)
    179                         observation, r, done, info = self.processor.process_step(observation, r, done, info)
    180                     for key, value in info.items():
--> 181                         if not np.isreal(value):
    182                             continue
    183                         if key not in accumulated_info:

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

复现代码 #

#import environment and visualization
import gym
from gym import wrappers
!pip install gym[classic_control]

#import Deep Learning api
import tensorflow as tf
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.layers import Dense, Flatten, Input, Embedding,Reshape
from tensorflow.keras.optimizers import Adam

#import rl agent library
!pip install gym
!pip install keras
!pip install keras-rl2

#data manipulation
import numpy as np
import pandas as pd
import random 
#0
env = gym.make('Taxi-v3')
env.reset()
actions=env.action_space.n
states=env.observation_space.n
#1
def build_model(states,actions):
  model=Sequential()
  model.add(Embedding(states,10, input_length=1))
  model.add(Reshape((10,)))
  model.add(Dense(32,activation='relu'))
  model.add(Dense(32,activation='relu'))
  model.add(Dense(actions,activation='linear'))
  return model
#2
import rl
from rl.agents import DQNAgent
from rl.policy import EpsGreedyQPolicy
from rl.memory import SequentialMemory

policy=EpsGreedyQPolicy()
memory=SequentialMemory(limit=100000,window_length=1)
#3
agent=DQNAgent(model=model1,memory=memory,policy=policy,nb_actions=actions,nb_steps_warmup=500, target_model_update=1e-2)
agent.compile(Adam(lr=0.001),metrics=['mae'])

agent.fit(env, nb_steps=1000000, visualize=False, verbose=1, nb_max_episode_steps=99,

解决方案 #

1. 修复info字典数组判断问题 #

报错根源是Taxi-v3环境返回的info字典中包含数组类型的值，而keras-rl2的fit方法中np.isreal(value)无法直接处理数组，推荐通过自定义处理器（Processor）过滤info中的非实数数组，无需修改库源码：

from rl.core import Processor

class TaxiProcessor(Processor):
    def process_step(self, observation, reward, done, info):
        # 只保留info中的标量实数
        filtered_info = {k: v for k, v in info.items() if np.isscalar(v) and np.isreal(v)}
        return observation, reward, done, filtered_info

初始化DQNAgent时指定该处理器：

agent=DQNAgent(
    model=model1,
    memory=memory,
    policy=policy,
    nb_actions=actions,
    nb_steps_warmup=500, 
    target_model_update=1e-2,
    processor=TaxiProcessor()
)

2. 扩展单轮训练步数 #

你已设置env._max_episode_steps=5000，但agent.fit中的nb_max_episode_steps=99会覆盖这个设置，需要删除该参数或调整为对应值：

agent.fit(
    env, 
    nb_steps=1000000, 
    visualize=False, 
    verbose=1, 
    log_interval=100000
    # 移除nb_max_episode_steps=99，或改为nb_max_episode_steps=5000
)

3. 修正模型初始化遗漏问题 #

代码中agent=DQNAgent(model=model1,...)的model1未定义，需先调用build_model生成模型：

model1 = build_model(states, actions)  # 补充该行
agent=DQNAgent(model=model1,memory=memory,policy=policy,nb_actions=actions,nb_steps_warmup=500, target_model_update=1e-2)

内容的提问来源于stack exchange，提问作者kaispace30098