ViZDoom DQN训练中Epsilon衰减过快问题求解

我用PyTorch训练基于深度强化学习的ViZDoom智能体，碰到Epsilon衰减太快的问题。之前用乘以epsilon_decay的方式，几步就衰减完了；改成按步数衰减后，速度还是超出预期。训练总步数设为600000步，但Epsilon在1000-2000步区间就完成了衰减。参考了论文《ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning》，以下是我的训练代码、智能体训练代码及参数配置，求优化建议：

训练代码 #

def train(self):

    # 训练相关代码


    # 计算epsilon
    self.steps += 1

    if self.steps >= self.epsilon_decay_start:
        decay_steps = self.steps - self.epsilon_decay_start
        total_decay_steps = self.epsilon_decay_end - self.epsilon_decay_start
        self.epsilon = max(self.epsilon_min, self.epsilon - decay_steps * ((self.epsilon - self.epsilon_min) / total_decay_steps))

    return self.epsilon

智能体训练代码 #

def train_agent(game, agent, actions, scenario, save_model, STEPS_TO_TRAIN, FRAME_REPEAT):
    max_steps = STEPS_TO_TRAIN
    steps_count = 0
    episode_count = 0
    episode_rewards = []
    epsilon_values = []
    max_reward = float('-inf')
    best_episode = -1

    print("Training")

    while steps_count < max_steps:
        recorded_episode = f"episode{episode_count}_rec.lmp"
        game.new_episode(recorded_episode)
    
        state = agent.preprocess(game.get_state().screen_buffer)
        total_reward = 0

        for step in it.count():
            if steps_count >= max_steps:
                break
            action = agent.select_action(state)
            reward = game.make_action(actions[action], FRAME_REPEAT)
            done = game.is_episode_finished()
            total_reward += reward
            next_state = agent.preprocess(game.get_state().screen_buffer) if not done else None
            agent.remember(state, action, next_state, reward)
            state = next_state
            steps_count += 1

            epsilon = agent.train()
            if epsilon is not None:
                epsilon_values.append(epsilon)
            if done:
                episode_rewards.append(total_reward)
                break

        if total_reward >= max_reward:
            max_reward = total_reward
            best_episode = episode_count

        if save_model:
            agent.save_model()

        episode_count += 1

参数配置 #

# Q-learning设置
LEARNING_RATE = 0.01
GAMMA = 0.99 
BUFFER_SIZE = 10000
EPSILON = 1.0
EPSILON_DECAY_START = 100000
EPSILON_DECAY_END = 200000
EPSILON_MIN = 0.1
BATCH_SIZE = 40
# 其他参数
FRAME_REPEAT = 4
RESOLUTION = (60, 45)
STEPS_TO_TRAIN = 600000

优化建议 #

1. 修复Epsilon衰减计算逻辑错误 #

当前公式用动态变化的self.epsilon计算衰减量，会导致衰减幅度混乱。正确的线性衰减应该基于初始epsilon值计算，确保从初始值平稳降到最小值：

def train(self):

    # 训练相关代码


    # 计算epsilon
    self.steps += 1

    if self.steps >= self.epsilon_decay_start:
        decay_steps = self.steps - self.epsilon_decay_start
        total_decay_steps = self.epsilon_decay_end - self.epsilon_decay_start
        # 基于初始epsilon值计算线性衰减比例
        decay_ratio = min(decay_steps / total_decay_steps, 1.0)
        self.epsilon = self.epsilon_initial - decay_ratio * (self.epsilon_initial - self.epsilon_min)
        self.epsilon = max(self.epsilon_min, self.epsilon)

    return self.epsilon

注意：需要在Agent初始化时保存self.epsilon_initial = 1.0，避免用动态变化的当前值计算。

2. 对齐参数配置与实际训练步数 #

你的参数中EPSILON_DECAY_START = 100000、EPSILON_DECAY_END = 200000，但你提到Epsilon在1000-2000步就衰减完成，说明参数配置和实际训练逻辑存在矛盾：

• 如果实际训练总步数是6000，调整衰减参数适配：

  EPSILON_DECAY_START = 1000  # 前1000步保持满探索
  EPSILON_DECAY_END = 5000    # 用4000步完成衰减到最小值
  

• 如果参数配置正确（总步数600000），检查self.steps的计数逻辑——是否每调用一次train()就加1，导致计数远快于实际游戏步数？

3. 降低Epsilon更新频率 #

当前每执行一步游戏就更新一次Epsilon，频率过高。可以改为仅在执行经验回放训练时更新Epsilon，和训练节奏对齐：

def train(self):
    epsilon_to_return = None
    # 只有当经验池足够大，执行训练时才更新epsilon
    if len(self.memory) > self.batch_size:
        # 执行经验回放训练逻辑
        # ...

        self.steps += 1
        if self.steps >= self.epsilon_decay_start:
            decay_steps = self.steps - self.epsilon_decay_start
            total_decay_steps = self.epsilon_decay_end - self.epsilon_decay_start
            decay_ratio = min(decay_steps / total_decay_steps, 1.0)
            self.epsilon = self.epsilon_initial - decay_ratio * (self.epsilon_initial - self.epsilon_min)
            self.epsilon = max(self.epsilon_min, self.epsilon)
        epsilon_to_return = self.epsilon
    return epsilon_to_return

4. 改用非线性衰减（可选） #

如果线性衰减不符合需求，可尝试指数衰减或余弦退火衰减，在训练初期保持高探索率，后期缓慢收敛：

• 指数衰减示例：

  if self.steps >= self.epsilon_decay_start:
      self.epsilon = max(self.epsilon_min, self.epsilon * 0.9999)
  

• 余弦退火衰减示例：

  import math
  
  if self.steps >= self.epsilon_decay_start:
      decay_steps = self.steps - self.epsilon_decay_start
      total_decay_steps = self.epsilon_decay_end - self.epsilon_decay_start
      if decay_steps <= total_decay_steps:
          cos_decay = 0.5 * (1 + math.cos(math.pi * decay_steps / total_decay_steps))
          self.epsilon = self.epsilon_min + (self.epsilon_initial - self.epsilon_min) * cos_decay
      else:
          self.epsilon = self.epsilon_min
  

内容的提问来源于stack exchange，提问作者MiosV