深度学习模型训练中Loss出现NaN问题求助

我用二进制数据实现深度学习模型，模型在其他二进制数据集上运行正常，但用自行采集的二进制数据训练时，Loss显示NaN值，准确率始终固定，推测问题出在数据而非模型。

模型代码 #

import tensorflow as tf
from tensorflow.keras.layers import \
    Dense, Dropout, GlobalAveragePooling1D, GlobalAveragePooling2D, Input, Activation, MaxPooling1D, MaxPooling2D, Conv1D, Conv2D, BatchNormalization, LSTM, Flatten, ELU, AveragePooling1D, Permute
from tensorflow.keras.initializers import Constant
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.regularizers import l2
from tensorflow.keras.models import Model
from keras import losses

import cvnn.layers as complex_layers
from tensorflow.keras.losses import Loss, categorical_crossentropy
from keras.optimizers import RMSprop
        
def createSB_cart(inp_shape, classes_num, emb_size=64, weight_decay=1e-6, classification=False):
    convArgs = dict(use_bias=False,
                    kernel_regularizer=l2(weight_decay)
                    )
    tf.device("gpu:1")
    model = tf.keras.models.Sequential()
    model.add(complex_layers.ComplexInput(input_shape=inp_shape))
    model.add(complex_layers.ComplexConv1D(256, 2, activation='cart_leaky_relu', padding='same', **convArgs))
    model.add(complex_layers.ComplexAvgPooling1D(2))
    model.add(complex_layers.ComplexDropout(rate=0.2))

    model.add(complex_layers.ComplexConv1D(256, 2, activation='cart_leaky_relu', padding='same', **convArgs))
    model.add(complex_layers.ComplexAvgPooling1D(2))
    model.add(complex_layers.ComplexDropout(rate=0.2))


    model.add(complex_layers.ComplexConv1D(128, 2, activation='cart_leaky_relu', padding='same',**convArgs))
    model.add(complex_layers.ComplexAvgPooling1D(2))
    model.add(complex_layers.ComplexDropout(rate=0.2))


    model.add(complex_layers.ComplexConv1D(128, 2, activation='cart_leaky_relu', padding='same', **convArgs))
    model.add(complex_layers.ComplexAvgPooling1D(2))
    model.add(complex_layers.ComplexDropout(rate=0.2))

    model.add(complex_layers.ComplexConv1D(16, 2, activation='cart_leaky_relu', padding='same', **convArgs))
    model.add(complex_layers.ComplexAvgPooling1D(2))
    model.add(complex_layers.ComplexFlatten())
    if classification:
        model.add(complex_layers.ComplexDense(classes_num, activation='convert_to_real_with_abs',**convArgs))
    else:
        model.add(complex_layers.ComplexDense(classes_num, activation='linear', **convArgs))

    return model

model.compile(loss=losses.mean_squared_error, metrics=["accuracy"], optimizer="RMSprop")

训练输出日志 #

Epoch 1/1000 101/101 [==============================] - ETA: 0s - loss: nan - accuracy: 0.5023
Epoch 1: val_accuracy improved from -inf to 0.49251, saving model to C:\Users\fafrin2\Downloads\radio\res_out\modelDir\IQ_model_our_day1_complex_after_fft_slices_5000_startIdx_0_stride_864_len_864_STFT_64.h5 101/101 [==============================] - 81s 792ms/step - loss: nan - accuracy: 0.5023 - val_loss: nan - val_accuracy: 0.4925 Epoch 2/1000 101/101 [==============================] - ETA: 0s - loss: nan - accuracy: 0.5019
Epoch 2: val_accuracy did not improve from 0.49251 101/101 [==============================] - 75s 743ms/step - loss: nan - accuracy: 0.5019 - val_loss: nan - val_accuracy: 0.4925

排查建议 #

• 检查自行采集的二进制数据是否存在NaN/无穷大值：读取数据后遍历验证，异常数值会直接导致Loss计算出现NaN。
• 对齐数据预处理流程：对比正常数据集的归一化/标准化操作，确认自行采集的数据是否做了相同的缩放（比如映射到[-1,1]区间），未归一化的大数值易引发梯度爆炸。
• 匹配任务与损失/指标：当前模型编译用了mean_squared_error（回归损失）但监控accuracy（分类指标），若为分类任务需替换损失为categorical_crossentropy，标签格式也要对应调整。
• 验证数据读取逻辑：确认二进制数据的解析方式（字节序、数据类型、维度）与正常数据集一致，比如复数数据是否正确拆分为实部+虚部，输入形状是否匹配模型的inp_shape参数。
• 降低学习率测试：暂时将RMSprop学习率调低至1e-5左右，若Loss不再出现NaN，说明数据分布差异引发梯度爆炸，需针对性优化预处理。

内容的提问来源于stack exchange，提问作者Fahmida Afrin