分批迭代数据集训练模型时遇IndexError错误的解决方法

我希望构建一个模型，按每批80000条数据迭代训练：先训练前80000条，再训练下一批80000条，以此类推。但执行代码时，在train_data[i]=X_train.iloc[i:i+length, :]处出现IndexError: list assignment index out of range错误，请问如何解决？

原代码如下：

length=80000
train_data=[]
train_tar=[]

for i in range (0, len(X_train), length):
    train_data[i]=X_train.iloc[i:i+length, :]
    train_tar[i]=Y_train.iloc[i:i+length, :]
    
   
    X_training, X_val, Y_training, Y_val = train_test_split(train_data[i], train_tar[i], test_size=0.40,   shuffle=False )
    
    scaler1= StandardScaler()
    X_training =scaler1.fit_transform(X_training[i])
    X_val[i]=scaler1.transform(X_val[i])
    X_test[i]=scaler1.transform(X_test[i])

    scaler2= StandardScaler()
    
    Y_training[i] =scaler2.fit_transform(Y_train[i])
    Y_val[i]=scaler2.transform(Y_val[i])
    Y_test[i]=scaler2.transform(Y_test[i])
    
    train_gen[i] = tf.keras.utils.timeseries_dataset_from_array(  X_training[i], Y_training[i],  sequence_length=160, sequence_stride=1, batch_size=256,sampling_rate=1,shuffle=False)
    
    val_gen[i] = tf.keras.utils.timeseries_dataset_from_array( X_val[i], Y_val[i], sequence_length=160, sequence_stride=1, batch_size=256,sampling_rate=1,shuffle=False)
    
    batch =train_gen
    inputs, target=batch
    input= inputs.shape[1], inputs.shape[2]
    print(input)
    
    
    def ann():
    
    
        model = Sequential()
        model.add(Dense(1000, input_shape=input))
        model.add(Dense(100))
        model.add(Flatten())
        model.add(Dense(2, activation='linear'))
        model.compile(optimizer=Adam(learning_rate = 1e-6), loss= 'mse', metrics=(['accuracy']))
        model.summary()
        return model

    model = ann()
    
    history=model.fit(train_gen[i], validation_data=val_gen[i], shuffle=True,epochs=10,verbose=1)

1. 列表索引赋值错误（核心问题） #

train_data和train_tar是空列表，不能直接通过train_data[i]赋值——列表只有在已有对应索引位置的元素时才能用索引修改，空列表没有任何索引位置，所以报错。应该用append()方法向列表添加元素：

train_data.append(X_train.iloc[i:i+length, :])
train_tar.append(Y_train.iloc[i:i+length, :])

2. 多余的数组索引访问 #

X_training、Y_training等是train_test_split返回的完整批次数据集（DataFrame或numpy数组），不需要再用X_training[i]访问索引——这会导致维度错误或索引越界。直接使用变量本身即可：

X_training = scaler1.fit_transform(X_training)
X_val = scaler1.transform(X_val)
X_test = scaler1.transform(X_test)  # 假设X_test是预先定义的完整测试集

3. 未初始化的列表 #

train_gen和val_gen没有预先定义为空列表，直接用train_gen[i]赋值会报错，需要先初始化后用append()添加元素：

# 初始化空列表
train_gen = []
val_gen = []

# 向列表添加生成器
train_gen.append(tf.keras.utils.timeseries_dataset_from_array(
    X_training, Y_training, 
    sequence_length=160, sequence_stride=1, 
    batch_size=256, sampling_rate=1, shuffle=False
))
val_gen.append(tf.keras.utils.timeseries_dataset_from_array(
    X_val, Y_val, 
    sequence_length=160, sequence_stride=1, 
    batch_size=256, sampling_rate=1, shuffle=False
))

4. 数据集维度与模型输入问题 #

• 不能直接解包列表类型的train_gen获取输入形状，应该取当前批次的生成器，再通过take(1)获取一个样本提取形状：

current_train_gen = train_gen[-1]
for inputs, target in current_train_gen.take(1):
    input_shape = (inputs.shape[1], inputs.shape[2])
print(input_shape)

• 模型中Flatten()层无需添加，Dense层会自动处理时间序列的二维输入（时间步+特征数），添加后会导致维度不匹配。

5. 其他问题修正 #

• Y_training[i] = scaler2.fit_transform(Y_train[i])误用了全局Y_train，应该用当前批次的Y_training；
• 时间序列数据训练时不建议开启shuffle=True，会打乱时序逻辑；
• 确保X_test、Y_test已预先定义，否则移除相关代码或补充定义。

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.optimizers import Adam
import tensorflow as tf

length = 80000
train_data = []
train_tar = []
train_gen = []
val_gen = []

# 假设X_train、Y_train、X_test、Y_test已预先定义
for i in range(0, len(X_train), length):
    # 添加批次数据到列表
    batch_X = X_train.iloc[i:i+length, :]
    batch_Y = Y_train.iloc[i:i+length, :]
    train_data.append(batch_X)
    train_tar.append(batch_Y)
    
    # 划分训练集与验证集
    X_training, X_val, Y_training, Y_val = train_test_split(batch_X, batch_Y, test_size=0.40, shuffle=False)
    
    # 标准化特征
    scaler1 = StandardScaler()
    X_training = scaler1.fit_transform(X_training)
    X_val = scaler1.transform(X_val)
    X_test = scaler1.transform(X_test)
    
    # 标准化标签
    scaler2 = StandardScaler()
    Y_training = scaler2.fit_transform(Y_training)
    Y_val = scaler2.transform(Y_val)
    Y_test = scaler2.transform(Y_test)
    
    # 创建时间序列生成器
    current_train_gen = tf.keras.utils.timeseries_dataset_from_array(
        X_training, Y_training,
        sequence_length=160, sequence_stride=1,
        batch_size=256, sampling_rate=1, shuffle=False
    )
    current_val_gen = tf.keras.utils.timeseries_dataset_from_array(
        X_val, Y_val,
        sequence_length=160, sequence_stride=1,
        batch_size=256, sampling_rate=1, shuffle=False
    )
    train_gen.append(current_train_gen)
    val_gen.append(current_val_gen)
    
    # 获取输入形状
    for inputs, target in current_train_gen.take(1):
        input_shape = (inputs.shape[1], inputs.shape[2])
    print(f"批次{i}输入形状：{input_shape}")
    
    # 定义模型
    def ann(input_shape):
        model = Sequential()
        model.add(Dense(1000, input_shape=input_shape))
        model.add(Dense(100))
        model.add(Dense(2, activation='linear'))
        model.compile(optimizer=Adam(learning_rate=1e-6), loss='mse', metrics=['accuracy'])
        model.summary()
        return model
    
    model = ann(input_shape)
    
    # 训练模型
    history = model.fit(current_train_gen, validation_data=current_val_gen, shuffle=False, epochs=10, verbose=1)

内容的提问来源于stack exchange，提问作者user_scc