为何Attention层无法提升多标签文本分类RCNN模型性能？

任务与数据集 #

正在开展多标签文本分类任务，数据集规模：

x_train shape: (8066, 3000)
x_test shape: (1729, 3000)
x_valid shape: (1573, 3000)

模型现状 #

实现的RCNN模型取得60的MiF分数，但添加自定义Attention层后分数仍为60。尝试调整批量大小、epoch数、dropout、批量归一化、隐藏单元尺寸等方案均无效；在RNN前后添加Attention层时性能反而下降，当前仅在Concatenate层后添加时性能与原模型持平。

不带Attention层的模型代码 #

import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from keras.layers import Embedding, Input, Bidirectional, LSTM
maxlen=3000
max_features= 50000
embedding_dims=100
input_length=maxlen

# Input for variable-length sequences of integers
inputs = keras.Input(shape=(maxlen,), dtype="int32")
# Embed each integer in a 100-dimensional vector
x = layers.Embedding(max_features,
                            embedding_dims,
                            weights=[embedding_matrix],
                            input_length=maxlen,
                            trainable=False)(inputs)
x = layers.Dropout(0.5)(x)

# Add 2 bidirectional LSTMs
x = layers.Bidirectional(layers.LSTM(128, return_sequences=True, recurrent_activation='sigmoid'))(x)
b = layers.Bidirectional(layers.LSTM(128, return_sequences=True, recurrent_activation='sigmoid'))(x)

c1 = layers.Conv1D(128, 1, activation='relu')(b)
A1 = layers.GlobalAveragePooling1D()(c1)
M1=  layers.GlobalMaxPooling1D()(c1)

c2 = layers.Conv1D(128, 2, activation='relu')(b)
A2 = layers.GlobalAveragePooling1D()(c2)
M2=  layers.GlobalMaxPooling1D()(c2)

c3 = layers.Conv1D(128, 3, activation='relu')(b)
A3 = layers.GlobalAveragePooling1D()(c3)
M3=  layers.GlobalMaxPooling1D()(c3)

c4 = layers.Conv1D(128, 4, activation='relu')(b)
A4 = layers.GlobalAveragePooling1D()(c4)
M4=  layers.GlobalMaxPooling1D()(c4)

c5 = layers.Conv1D(128, 5, activation='relu')(b)
A5 = layers.GlobalAveragePooling1D()(c5)
M5 =  layers.GlobalMaxPooling1D()(c5)


concat = layers.concatenate([A1, M1, A2, M2, A3, M3, A4, M4, A5, M5])

outputs = layers.Dense(50, activation="sigmoid")(concat)
model = keras.Model(inputs, outputs)

自定义Attention层代码 #

from tensorflow.keras import backend as K
from tensorflow.keras import initializers, regularizers, constraints
from tensorflow.keras.layers import Layer


def dot_product(x, kernel):
    """
    Wrapper for dot product operation, in order to be compatible with both
    Theano and Tensorflow
    Args:
        x (): input
        kernel (): weights
    Returns:
    """
    if K.backend() == 'tensorflow':
        return K.squeeze(K.dot(x, K.expand_dims(kernel)), axis=-1)
    else:
        return K.dot(x, kernel)


class Attention(Layer):
    def __init__(self,
                 kernel_regularizer=None, bias_regularizer=None,
                 W_constraint=None, b_constraint=None,
                 bias=True,
                 return_attention=False,
                 **kwargs):
        """
        Keras Layer that implements an Attention mechanism for temporal data.
        Supports Masking.
        Follows the work of Raffel et al. [https://arxiv.org/abs/1512.08756]
        # Input shape
            3D tensor with shape: `(samples, steps, features)`.
        # Output shape
            2D tensor with shape: `(samples, features)`.
        :param kwargs:
        Just put it on top of an RNN Layer (GRU/LSTM/SimpleRNN) with return_sequences=True.
        The dimensions are inferred based on the output shape of the RNN.
        Note: The layer has been tested with Keras 1.x
        Example:
            model.add(LSTM(64, return_sequences=True))
            model.add(Attention())
            # next add a Dense layer (for classification/regression) or whatever...
        """
        self.supports_masking = True
        self.init = initializers.get('glorot_uniform')

        self.W_regularizer = regularizers.get(kernel_regularizer)
        self.b_regularizer = regularizers.get(bias_regularizer)

        self.W_constraint = constraints.get(W_constraint)
        self.b_constraint = constraints.get(b_constraint)

        self.bias = bias
        self.return_attention = return_attention
        super(Attention, self).__init__(**kwargs)

    def build(self, input_shape):
        assert len(input_shape) == 3

        self.W = self.add_weight(shape=(input_shape[-1],),
                                 initializer=self.init,
                                 name='{}_W'.format(self.name))

        if self.bias:
            self.b = self.add_weight(shape=(input_shape[1],),
                                     initializer='zeros',
                                     name='{}_b'.format(self.name),
                                     regularizer=self.b_regularizer,
                                     constraint=self.b_constraint)

        self.built = True

    def compute_mask(self, inputs, mask=None):
        # do not pass the mask to the next layers
        if self.return_attention:
            return [None, None]
        return None

    def call(self, x, mask=None):
        eij = dot_product(x, self.W)

        if self.bias:
            eij += self.b

        eij = K.tanh(eij)

        a = K.exp(eij)

        # apply mask after the exp. will be re-normalized next
        if mask is not None:
            # Cast the mask to floatX to avoid float64 upcasting in theano
            a *= K.cast(mask, K.floatx())

        # in some cases especially in the early stages of training the sum may be almost zero
        # and this results in NaN's. A workaround is to add a very small positive number ε to the sum.
        # a /= K.cast(K.sum(a, axis=1, keepdims=True), K.floatx())
        a /= K.cast(K.sum(a, axis=1, keepdims=True) + K.epsilon(), K.floatx())

        a = K.expand_dims(a)
        weighted_input = x * a
        result = K.sum(weighted_input, axis=1)

        if self.return_attention:
            return [result, a]
        return result

    def compute_output_shape(self, input_shape):
        if self.return_attention:
            return [(input_shape[0], input_shape[-1]),
                    (input_shape[0], input_shape[1])]
        else:
            return input_shape[0], input_shape[-1]
    def get_config(self):
        config = super().get_config()
        return config        

带Attention层的模型代码 #

import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from keras.layers import Embedding, Input, Bidirectional, LSTM
maxlen=3000
max_features= 50000
embedding_dims=100
input_length=maxlen

# Input for variable-length sequences of integers
inputs = keras.Input(shape=(maxlen,), dtype="int32")
# Embed each integer in a 100-dimensional vector
x = layers.Embedding(max_features,
                            embedding_dims,
                            weights=[embedding_matrix],
                            input_length=maxlen,
                            trainable=False)(inputs)
x = layers.Dropout(0.5)(x)

# Add 2 bidirectional LSTMs
x = layers.Bidirectional(layers.LSTM(128, return_sequences=True, recurrent_activation='sigmoid'))(x)
b = layers.Bidirectional(layers.LSTM(128, return_sequences=True, recurrent_activation='sigmoid'))(x)

c1 = layers.Conv1D(128, 1, activation='relu')(b)
A1 = layers.GlobalAveragePooling1D()(c1)
M1=  layers.GlobalMaxPooling1D()(c1)

c2 = layers.Conv1D(128, 2, activation='relu')(b)
A2 = layers.GlobalAveragePooling1D()(c2)
M2=  layers.GlobalMaxPooling1D()(c2)

c3 = layers.Conv1D(128, 3, activation='relu')(b)
A3 = layers.GlobalAveragePooling1D()(c3)
M3=  layers.GlobalMaxPooling1D()(c3)

c4 = layers.Conv1D(128, 4, activation='relu')(b)
A4 = layers.GlobalAveragePooling1D()(c4)
M4=  layers.GlobalMaxPooling1D()(c4)

c5 = layers.Conv1D(128, 5, activation='relu')(b)
A5 = layers.GlobalAveragePooling1D()(c5)
M5 =  layers.GlobalMaxPooling1D()(c5)


concat = layers.concatenate([A1, M1, A2, M2, A3, M3, A4, M4, A5, M5])
reshape_layer = layers.Reshape([1, 1280]) (concat)

att = Attention()(reshape_layer)
batch= layers.BatchNormalization()(att)
outputs = layers.Dense(50, activation="sigmoid")(batch)
model = keras.Model(inputs, outputs)

解决建议 #

1. 修正Concatenate层后Attention的使用逻辑 #

当前将concat（2D张量：(batch_size, 1280)）reshape为3D张量后加Attention，本质上是对仅有的1个"时间步"做注意力，完全没有意义——注意力机制需要多个序列步才能学习权重差异。直接去掉这部分操作，或者把Attention放在真正有序列信息的地方：比如Bidirectional LSTM的输出之后，也就是b张量（3D：(batch_size, 3000, 256)）之后，替换掉后续的Conv+Pool组合，或者和Conv分支并行。

示例调整：

# 原LSTM输出后加Attention
b = layers.Bidirectional(layers.LSTM(128, return_sequences=True, recurrent_activation='sigmoid'))(x)
att_output = Attention()(b)  # 得到2D张量(batch_size,256)

# 保留原Conv分支
c1 = layers.Conv1D(128, 1, activation='relu')(b)
A1 = layers.GlobalAveragePooling1D()(c1)
M1=  layers.GlobalMaxPooling1D()(c1)
# ... 其他Conv分支省略

# 把Attention输出和Conv分支的Pool结果拼接
concat = layers.concatenate([att_output, A1, M1, A2, M2, A3, M3, A4, M4, A5, M5])
outputs = layers.Dense(50, activation="sigmoid")(concat)

2. 检查Attention层的有效性 #

当前自定义Attention是单权重向量的全局注意力，对于长序列（3000步）可能效果有限。可以尝试：

• 改用多头注意力（TensorFlow内置的layers.MultiHeadAttention），更适合捕捉序列中不同位置的依赖关系；
• 在Attention层后添加可训练的映射层，比如Dense(256, activation='relu')，增强特征转换能力；
• 给Attention层添加正则化（比如kernel_regularizer=regularizers.l2(1e-4)），防止过拟合。

3. 优化原模型的瓶颈 #

原模型的MiF分数仅60，说明模型本身还有提升空间，先优化基础模型再考虑Attention：

• 尝试将Embedding层设为trainable=True，让预训练嵌入适应任务数据；
• 减少LSTM层数或单元数，当前2层双向LSTM共512单元，加上多个Conv分支，可能在小数据集（8k训练样本）上过拟合；
• 调整损失函数：多标签分类建议用BinaryCrossentropy，确保编译时设置loss=tf.keras.losses.BinaryCrossentropy()；
• 增加学习率调度，比如用ReduceLROnPlateau回调，在验证集性能停滞时降低学习率。

4. 分析Attention的权重分布 #

启用Attention层的return_attention=True，输出注意力权重，查看是否集中在无意义的位置（比如padding部分），如果是，需要确保输入序列的mask正确传递——检查Embedding层是否设置了mask_zero=True（如果输入用0做padding的话）。

内容的提问来源于stack exchange，提问作者Raz