分类序列预测神经网络无学习效果的问题排查及优化

问题概述 #

• 任务：基于分类数据序列预测下一个元素，共3类，随机预测准确率为1/3
• 现状：模型预测准确率始终接近1/3，无法捕捉重复固定模式的规律
• 背景：神经网络领域新手，对输入配置、模型结构的合理性存疑

尝试过的输入配置 #

• 方案1：x为序列索引（共N个元素），y为对应元素，生成N个样本
• 方案2：x为除最后一个元素的完整序列向量，y为除第一个元素的完整序列向量，仅生成1个样本
• 方案3：x为长度递增的序列片段（x1含第1个元素，x2含前2个元素，以此类推），y为每个x对应的下一个元素，生成N-1个样本（采用此方案编写初始代码）

初始代码与问题 #

采用方案3的代码如下：

import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.preprocessing.sequence import pad_sequences

data = ["A", "B", "C", "A", "B", "C", "A", "B", "C", "A", "B", "C", "A", "B", "C"]
prev_history = []
correct_guess = 0

for i in range(1,100):
    input_size = len(data)
    vocab = ["A", "B", "C"]
    layer = keras.layers.StringLookup(vocabulary=vocab)

    x_train = np.array(layer(data))
    x_train = x_train - 1

    x_sequences = []
    y_sequences = []
    for i in range(1, input_size):
        x_sequences.append(x_train[:i])
        y_sequences.append(x_train[i])

    x_sequences[-1] = np.concatenate((x_sequences[-1], [-1]), axis=0)
    x_train = keras.utils.pad_sequences(x_sequences, padding='post', value=-1)
    y_train = np.array(y_sequences).transpose()

    model = keras.Sequential([
        keras.layers.Dense(8, activation='relu', input_dim=input_size),
        keras.layers.Dense(8, activation='relu'),
        keras.layers.Dense(3, activation='softmax')
    ])

    model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

    model.fit(x_train, y_train, epochs=1, batch_size=1, shuffle=False)
    
    if prev_history != []:
        if data[-1] == prev_history[-1]:
            correct_guess += 1
        if len(prev_history) > 0:
            correct_guess_rate = correct_guess/(len(prev_history))

    x_pred = np.array(layer(data))
    x_pred = x_pred - 1
    x_pred = x_pred.reshape(1, input_size)
    predchance = model.predict(x_pred)
    max_n = np.argmax(predchance)
    pred = layer.get_vocabulary()[max_n+1]
    prev_history.append(pred)

    choices = ["A", "B", "C"]
    data.append(choices[input_size % len(choices)])

问题表现 #

• 尝试增加序列长度、网络层数、神经元数量、epoch数、更换模型等操作，均无法提升准确率，始终接近随机水平
• 无法识别明显的重复模式，输入配置和batch_size设置的合理性存疑

解决方案与更新 #

修改输入配置为固定长度（3）的序列片段作为输入，对应下一个元素为目标，并改用SimpleRNN模型，代码如下：

import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow import keras

data = ["A", "B", "C", "A", "B", "C", "A", "B", "C", "A", "B", "C", "A", "B", "C", "A", "B", "C", "A", "B", "C"]
prev_history = []
correct_guess = 0

for i in range(1,100):
    input_size = len(data)
    vocab = ["A", "B", "C"]
    layer = keras.layers.StringLookup(vocabulary=vocab)
    data_u = layer(data)
    data_u = data_u - 1
    seq_length = 3

    dataset = keras.utils.timeseries_dataset_from_array(
        data_u.numpy(),
        targets = data_u[seq_length:],
        sequence_length = seq_length,
        batch_size = 2
    )

    model = keras.Sequential([
        keras.layers.SimpleRNN(32, activation='relu', input_shape=[None,1]),
        keras.layers.Dense(3, activation='softmax')
    ])

    model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

    model.fit(dataset, epochs=10, shuffle=True)
    
    if prev_history != []:
        if data[-1] == prev_history[-1]:
            correct_guess += 1
        if len(prev_history) > 0:
            correct_guess_rate = correct_guess/(len(prev_history))

    x_pred = data_u[-seq_length:]
    x_pred = tf.reshape(x_pred, [1,seq_length])
    predchance = model.predict(x_pred)
    max_n = np.argmax(predchance)
    pred = layer.get_vocabulary()[max_n+1]
    prev_history.append(pred)

    choices = ["A", "B", "C"]
    data.append(choices[input_size % len(choices)])

效果 #

• 预测准确率显著优于随机选择，能够稳定捕捉序列中的重复模式
• 更换不同模式测试，均能保持良好的预测表现

内容的提问来源于stack exchange，提问作者Segala