多标签分类任务中如何设置损失函数惩罚模型的全零预测行为？

核心问题解答 #

1. weighted_cross_entropy_with_logits对输出层激活函数的影响 #

tf.nn.weighted_cross_entropy_with_logits 内部已经内置了sigmoid运算逻辑，会直接对输入的原始logits做sigmoid转换再计算交叉熵，不需要用户在输出层额外添加sigmoid激活。如果保留输出层的sigmoid，相当于对输出做了两次sigmoid映射，会导致数值压缩过度、梯度消失，模型无法正常收敛。

2. 是否需要移除输出层激活函数？ #

是，必须将输出层的activation='sigmoid'改为activation=None，直接输出原始logits值即可。后续预测时如果需要得到0-1之间的概率值，手动调用sigmoid对输出logits做转换即可。

3. 正样本错分惩罚的参数配置 #

通过损失函数的pos_weight参数配置正样本的损失权重，取值规则为：

pos_weight = 负样本总数量 / 正样本总数量

你的场景下正样本占比约为1/30，因此pos_weight可以设置为29~30之间，即可将正样本错分的损失放大29倍，避免模型为了整体准确率倾向于输出全零结果，有效提升召回率。

修改后可运行代码 #

import numpy as np
import tensorflow as tf
from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model


OUTPUT_NODES = 30
POS_WEIGHT = 29  # 正样本损失权重
np.random.seed(0)

# 自定义加权交叉熵损失函数
def weighted_binary_crossentropy(y_true, y_pred):
    return tf.nn.weighted_cross_entropy_with_logits(
        labels=y_true,
        logits=y_pred,
        pos_weight=POS_WEIGHT
    )

def get_dataset():
    """
    Get a dataset of X and y. This is a learnable problem as there is some signal in the features. 10% of the time, a
    positive-output's index will also have a positive feature for that index
    :return: X and y data for training
    """
    n_observations = 30000
    y = np.random.rand(n_observations, OUTPUT_NODES)
    y = (y <= (1 / OUTPUT_NODES)).astype(int)  # Makes a sparse output where there is roughly 1 positive label: ((1 / OUTPUT_NODES) * OUTPUT_NODES ≈ 1)

    X = np.zeros((n_observations, OUTPUT_NODES))
    for i in range(len(y)):
        for j, feature in enumerate(y[i]):
            if feature == 1:
                X[i][j] = 1 if np.random.rand(1) > 0.9 else 0  # Makes the input features more noisy
                # X[i][j] = 1  # Using this instead will make the model perform very well

    return X, y


def create_model():
    input_layer = Input(shape=(OUTPUT_NODES, ))
    dense1 = Dense(100, activation='relu')(input_layer)
    dense2 = Dense(100, activation='relu')(dense1)
    # 移除sigmoid激活，输出原始logits
    output_layer = Dense(30, activation=None)(dense2)

    model = Model(inputs=input_layer, outputs=output_layer)
    # 替换损失函数为自定义的加权交叉熵
    model.compile(optimizer='adam', loss=weighted_binary_crossentropy, metrics=['Recall'])

    return model


def main():
    X, y = get_dataset()
    model = create_model()
    model.fit(X, y, epochs=10, batch_size=10)

    X_pred = np.random.randint(0, 2, (100, OUTPUT_NODES))
    y_pred_logits = model.predict(X_pred)
    # 手动添加sigmoid转换为概率
    y_pred = tf.sigmoid(y_pred_logits).numpy()

    print(X_pred)
    print(y_pred.round(1))


if __name__ == '__main__':
    main()

内容的提问来源于stack exchange，提问作者Jason p