基于RNN的乙醇浓度时序预测：测试集R2异常偏低问题排查

核心问题诊断 #

你的代码存在多个关键逻辑错误，直接导致模型训练和评估失效，主要问题包括：

• 特征选择时的数据泄露（用测试集标签计算训练集特征相关性）
• 输入序列与预测标签完全不匹配
• 归一化未覆盖标签，且统计量计算错误
• 模型输出与目标维度不匹配
• 评估逻辑严重偏离任务目标（用均值代替时序真实值）

修复步骤与修正后的代码 #

1. 修正特征选择逻辑

必须用训练集自身的乙醇浓度计算特征相关性，避免数据泄露：

# 原错误：用测试集y计算训练集特征相关性
# 修正：使用训练集的因变量
X = df_train.iloc[:,4:37]
y = df_train.iloc[:,3]
correlations = X.corrwith(y)
correlations = correlations.abs().sort_values(ascending=False)
top_5_conditions = correlations.head(5).index.tolist()

2. 正确构建时序数据（输入-标签对应）

任务是预测1-8小时的乙醇浓度，正确的时序构建应该是：每个样本-实验的完整8小时生物条件序列，对应相同时间窗口的乙醇浓度序列。这里采用序列到序列的方式匹配输入与标签：

def create_sequences_and_labels(data):
    sequences = []
    labels = []
    for (sample, exp), group in data.groupby(['Sample', 'Experiment']):
        # 每个样本-实验对应完整8小时数据
        seq = group[top_5_conditions].values
        label = group['Ethanol_Concentration'].values
        sequences.append(seq)
        labels.append(label)
    return np.array(sequences), np.array(labels)

train_sequences, train_labels = create_sequences_and_labels(train_data)
test_sequences, test_labels = create_sequences_and_labels(test_data)

3. 正确归一化特征与标签

分别对特征和标签做归一化，且测试集严格使用训练集的统计量，避免数据泄露：

# 归一化特征（按特征维度计算统计量）
feat_mean = train_sequences.mean(axis=(0,1))
feat_std = train_sequences.std(axis=(0,1))
train_sequences = (train_sequences - feat_mean) / feat_std
test_sequences = (test_sequences - feat_mean) / feat_std

# 归一化标签（乙醇浓度）
label_mean = train_labels.mean()
label_std = train_labels.std()
train_labels = (train_labels - label_mean) / label_std
test_labels = (test_labels - label_mean) / label_std

4. 调整模型结构匹配任务

因为要预测8个时间步的浓度，模型输出维度需设为8，同时增加LSTM层提升时序建模能力：

model = Sequential()
model.add(LSTM(units=64, return_sequences=True, input_shape=(8, 5)))
model.add(LSTM(units=32))
model.add(Dense(units=8))  # 输出8个时间步的预测值
model.compile(loss=root_mean_squared_error, optimizer=RMSprop(lr=0.001))

5. 修正评估逻辑

计算整体时序的R2分数，同时按样本-实验输出详细结果，不再用均值代替真实时序值：

from sklearn.metrics import r2_score

# 预测并反归一化回到原始尺度
predictions = model.predict(test_sequences)
predictions = predictions * label_std + label_mean
test_labels = test_labels * label_std + label_mean

# 计算整体R2（展平所有时间步的真实值与预测值）
overall_r2 = r2_score(test_labels.flatten(), predictions.flatten())
print(f'整体R2分数: {overall_r2.round(4)}')

# 按样本-实验输出详细结果
for i, ((sample, exp), _) in enumerate(test_data.groupby(['Sample', 'Experiment'])):
    sample_r2 = r2_score(test_labels[i], predictions[i])
    print(f'Sample: {sample}, Experiment: {exp}')
    print(f'真实浓度: {test_labels[i].round(4)}')
    print(f'预测浓度: {predictions[i].round(4)}')
    print(f'该样本R2: {sample_r2.round(4)}\n')

完整修正代码 #

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.layers import Dense, LSTM
from keras import backend as K
from keras.optimizers import RMSprop
from sklearn.metrics import r2_score

# 自定义损失函数
def root_mean_squared_error(y_true, y_pred):
    return K.sqrt(K.mean(K.square(y_pred - y_true)))

# 加载训练和测试数据
df_train = pd.read_csv('/content/trainset8.csv')
df_test = pd.read_csv('/content/testset8.csv')

# 选择与乙醇浓度相关性最高的5项生物条件（使用训练集自身数据）
X = df_train.iloc[:,4:37]
y = df_train.iloc[:,3]
correlations = X.corrwith(y)
correlations = correlations.abs().sort_values(ascending=False)
top_5_conditions = correlations.head(5).index.tolist()

# 选择对应列
cols_to_keep = ["Time", "Sample", "Experiment", "Ethanol_Concentration"] + top_5_conditions
train_data = df_train[cols_to_keep]
test_data = df_test[cols_to_keep]

# 为训练集和测试集创建序列与标签（每个样本-实验对应完整8小时数据）
def create_sequences_and_labels(data):
    sequences = []
    labels = []
    for (sample, exp), group in data.groupby(['Sample', 'Experiment']):
        seq = group[top_5_conditions].values
        label = group['Ethanol_Concentration'].values
        sequences.append(seq)
        labels.append(label)
    return np.array(sequences), np.array(labels)

train_sequences, train_labels = create_sequences_and_labels(train_data)
test_sequences, test_labels = create_sequences_and_labels(test_data)

# 归一化特征（按特征维度计算统计量）
feat_mean = train_sequences.mean(axis=(0,1))
feat_std = train_sequences.std(axis=(0,1))
train_sequences = (train_sequences - feat_mean) / feat_std
test_sequences = (test_sequences - feat_mean) / feat_std

# 归一化标签（乙醇浓度）
label_mean = train_labels.mean()
label_std = train_labels.std()
train_labels = (train_labels - label_mean) / label_std
test_labels = (test_labels - label_mean) / label_std

# 划分训练集和验证集
train_X, val_X, train_y, val_y = train_test_split(train_sequences, train_labels, test_size=0.2, random_state=42)

# 创建LSTM模型（适配序列到序列预测）
model = Sequential()
model.add(LSTM(units=64, return_sequences=True, input_shape=(8, 5)))
model.add(LSTM(units=32))
model.add(Dense(units=8))
model.compile(loss=root_mean_squared_error, optimizer=RMSprop(lr=0.001))

# 训练模型
history = model.fit(train_X, train_y, epochs=50, batch_size=32, validation_data=(val_X, val_y))

# 在测试数据上评估模型
predictions = model.predict(test_sequences)
# 反归一化回到原始尺度
predictions = predictions * label_std + label_mean
test_labels = test_labels * label_std + label_mean

# 计算整体R2分数
overall_r2 = r2_score(test_labels.flatten(), predictions.flatten())
print(f'整体R2分数: {overall_r2.round(4)}')

# 按样本-实验输出详细结果
print("\n--- 样本-实验详细预测结果 ---")
for i, ((sample, exp), _) in enumerate(test_data.groupby(['Sample', 'Experiment'])):
    sample_r2 = r2_score(test_labels[i], predictions[i])
    print(f'Sample: {sample}, Experiment: {exp}')
    print(f'真实乙醇浓度: {test_labels[i].round(4)}')
    print(f'预测乙醇浓度: {predictions[i].round(4)}')
    print(f'该样本R2分数: {sample_r2.round(4)}\n')

额外优化建议 #

• 尝试单步预测：用前k小时的生物条件预测第k+1小时的乙醇浓度，这种任务设置更符合LSTM的常见使用场景，可能提升效果
• 增加早停机制：使用keras.callbacks.EarlyStopping避免过拟合
• 调整模型超参数：比如LSTM单元数量、学习率、批大小，或增加Dropout层抑制过拟合
• 验证数据时序连续性：确保每个样本-实验的时间序列是按时间顺序排列的，避免乱序导致模型学习无效规律

内容的提问来源于stack exchange，提问作者Mita