如何在深度强化学习模型中输出数组形式的多智能体动作？

针对你开发的自定义多智能体DQN环境（action_space为列表，如[2,4,3,2,1]，每个元素对应单个智能体的动作数），原单输出模型无法适配的问题，可通过以下两种方案解决：

方案一：多输出头模型（推荐） #

利用Keras函数式API构建多输出模型，每个智能体对应独立的输出层，避免动作空间爆炸，逻辑更清晰。

1. 构建多输出模型 #

放弃Sequential单输出结构，改用函数式API实现共享或独立中间层，最后为每个智能体添加对应动作数的输出层：

from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Dense

action_space = [2,4,3,2,1]  # 各智能体的动作数
states = env.observation_space.shape

def build_multi_output_model(states, action_space):
    # 输入层
    input_layer = Input(shape=states)
    # 共享中间层（也可为每个智能体单独设置层，根据需求调整）
    x = Dense(24, activation='relu')(input_layer)
    x = Dense(24, activation='relu')(x)
    # 为每个智能体创建输出层
    outputs = []
    for num_actions in action_space:
        outputs.append(Dense(num_actions, activation='linear', name=f"agent_{len(outputs)}_output")(x))
    # 构建多输出模型
    model = Model(inputs=input_layer, outputs=outputs)
    return model

model = build_multi_output_model(states, action_space)
model.compile(optimizer='adam', loss='mse')

2. 自定义训练逻辑（替代keras-rl的DQNAgent） #

由于keras-rl的DQNAgent默认只支持单输出，需手动实现经验回放、Q值计算和训练循环：

import numpy as np
from collections import deque

# 经验回放缓冲区
memory = deque(maxlen=50000)
gamma = 0.95  # 折扣因子
epsilon = 1.0  # 探索率
epsilon_min = 0.01
epsilon_decay = 0.995

def remember(state, action_list, reward, next_state, done):
    memory.append((state, action_list, reward, next_state, done))

def act(state):
    action_list = []
    if np.random.rand() <= epsilon:
        # 随机选择动作
        for num_actions in action_space:
            action_list.append(np.random.choice(num_actions))
    else:
        # 模型预测最优动作
        q_values = model.predict(state[np.newaxis, :], verbose=0)
        for q in q_values:
            action_list.append(np.argmax(q))
    return action_list

def replay(batch_size):
    global epsilon
    if len(memory) < batch_size:
        return
    batch = np.random.choice(len(memory), batch_size, replace=False)
    for i in batch:
        state, action_list, reward, next_state, done = memory[i]
        # 计算每个智能体的目标Q值
        target_q_list = model.predict(state[np.newaxis, :], verbose=0)
        next_q_list = model.predict(next_state[np.newaxis, :], verbose=0)
        
        for idx in range(len(action_space)):
            if done:
                target_q_list[idx][0][action_list[idx]] = reward
            else:
                target_q_list[idx][0][action_list[idx]] = reward + gamma * np.amax(next_q_list[idx][0])
        
        # 更新模型
        model.fit(state[np.newaxis, :], target_q_list, epochs=1, verbose=0)
    
    if epsilon > epsilon_min:
        epsilon *= epsilon_decay

3. 训练流程示例 #

batch_size = 32
episodes = 1000

for e in range(episodes):
    state = env.reset()
    done = False
    total_reward = 0
    while not done:
        action_list = act(state)
        next_state, reward, done, _ = env.step(action_list)
        total_reward += reward
        remember(state, action_list, reward, next_state, done)
        state = next_state
        replay(batch_size)
    print(f"Episode: {e+1}/{episodes}, Total Reward: {total_reward}, Epsilon: {epsilon:.2f}")

方案二：扁平化动作空间（仅适用于小规模场景） #

将所有智能体的动作组合转化为单个扁平化的动作空间，例如[2,4,3,2,1]可转化为2*4*3*2*1=48个唯一动作索引，每个索引对应一组智能体动作组合。

优缺点 #

• 优点：可复用原有单输出DQN代码，无需修改模型结构
• 缺点：动作空间随智能体数量/动作数指数增长，当智能体多或动作数大时，模型训练效率极低甚至不可行

动作转换示例 #

def action_list_to_idx(action_list, action_space):
    idx = 0
    multiplier = 1
    for num_actions, action in zip(reversed(action_space), reversed(action_list)):
        idx += action * multiplier
        multiplier *= num_actions
    return idx

def idx_to_action_list(idx, action_space):
    action_list = []
    remaining = idx
    for num_actions in action_space:
        action = remaining % num_actions
        action_list.append(action)
        remaining = remaining // num_actions
    return action_list

之后即可用原有单输出模型训练，将动作索引传入env.step前转换为动作列表即可。

内容的提问来源于stack exchange，提问作者Zhenyu He