LSTM时序数据预处理出现预期索引存在的KeyError问题

为提升LSTM模型性能，对时序数据做了归一化、序列生成，并按年份划分训练测试集。第一个geoid10对应的数据处理正常，但处理第二个时抛出KeyError:11，已确认该数据存在且长度符合要求。

相关代码 #

test = {}
tracks = incident_data["geoid10"].unique().tolist()
train_x, train_y = creatsequence(
    incident_data[incident_data["geoid10"] == tracks[0]], 4
)
tracks.pop(0)
for track in tracks:
    test_data = incident_data[incident_data["geoid10"] == track]
    if test_data[test_data["Year"] < 2016].shape[0] > 4:
        trainx, trainy = creatsequence(test_data[test_data["Year"] < 2016], 4)
        train_x = np.concatenate((train_x, trainx))
        train_y = np.concatenate((train_y, trainy))
        if test_data[test_data["Year"] >= 2015].shape[0] > 4:
            test_x, test_y = creatsequence(test_data[test_data["Year"] >= 2015], 4)
            test[track] = {"X": test_x, "y": test_y}

def creatsequence(data, length):
    x = []
    y = []
    for column in ["All Other Thefts_y", "Simple Assault_y", "Theft From Motor Vehicle_y"]:
        data[column] = normalizeSeries(data[column], 4)
    for i in range(len(data) - length):
        x.append(data.drop(columns=['geoid10',
        'Year','Quarter'])[i:i+length])
        y.append(np.array(data[["All Other Thefts_y", "Simple Assault_y", "Theft From Motor Vehicle_y"]])[i+length])
        
    return(np.array(x), np.array(y))

def normalizeSeries(x, priorDays):

    new_x = []
    
    start = len(x) - 1
    while start > 0:
        print(x)
        if start - priorDays > 4:
            subarray = x[start - 2 * priorDays : start - priorDays]
        else:
            subarray = x[0:4]
            
        max_value = max([max(subarray),0.001])
        for i in range(4):
            value = x[start]
            new_x.append(value / max_value)
            start -= 1

    return new_x

报错信息 #

KeyError: 11

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
/tmp/ipykernel_12/1047915744.py in <cell line: 11>()
     12     test_data = incident_data[incident_data["geoid10"] == track]
     13     if test_data[test_data["Year"] < 2016].shape[0] > 4:
---&gt; 14         trainx, trainy = creatsequence(test_data[test_data["Year"] < 2016], 4)
     15         train_x = np.concatenate((train_x, trainx))
     16         train_y = np.concatenate((train_y, trainy))

/tmp/ipykernel_12/2018859376.py in creatsequence(data, length)
      8     y = []
      9     for column in ["All Other Thefts_y", "Simple Assault_y", "Theft From Motor Vehicle_y"]:
---&gt; 10         data[column] = normalizeSeries(data[column], 4)
     11     for i in range(len(data) - length):
     12         x.append(data.drop(columns=['geoid10',

/tmp/ipykernel_12/2642226622.py in normalizeSeries(x, priorDays)
     17         max_value = max([max(subarray),0.001])
     18         for i in range(4):
---&gt; 19             value = x[start]
     20             new_x.append(value / max_value)
     21             start -= 1

~/.cache/pypoetry/virtualenvs/python-kernel-OtKFaj5M-py3.9/lib/python3.9/site-packages/pandas/core/series.py in __getitem__(self, key)
    979 
    980         elif key_is_scalar:
--&gt; 981             return self._get_value(key)
    982 
    983         if is_hashable(key):

~/.cache/pypoetry/virtualenvs/python-kernel-OtKFaj5M-py3.9/lib/python3.9/site-packages/pandas/core/series.py in _get_value(self, label, takeable)
   1087 
   1088         # Similar to Index.get_value, but we do not fall back to positional
-&gt; 1089         loc = self.index.get_loc(label)
   1090         return self.index._get_values_for_loc(self, loc, label)
   1091 

~/.cache/pypoetry/virtualenvs/python-kernel-OtKFaj5M-py3.9/lib/python3.9/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   3802                 return self._engine.get_loc(casted_key)
   3803             except KeyError as err:
-&gt; 3804                 raise KeyError(key) from err
   3805             except TypeError:
   3806                 # If we have a listlike key, _check_indexing_error will raise

问题原因与解决方法 #

核心原因 #

报错出现在normalizeSeries函数的value = x[start]行：这里的x是Pandas Series，你用整数start直接取值时，Pandas会把它当作标签索引而非位置索引。第二个geoid10的数据经过筛选后，索引不是连续的整数序列，因此找不到标签为11的元素。

同时，当前normalizeSeries的循环逻辑会生成长度为4*len(x)的列表，远大于原数据长度，后续赋值回原Series时还会触发长度不匹配的错误。

修复方案 #

1. 改用位置索引取值：将normalizeSeries中的取值代码改为.iloc，明确按位置取元素：

   value = x.iloc[start]
   

2. 修正归一化函数的长度问题：调整循环逻辑，确保输出的new_x长度和输入x一致，示例如下：

   def normalizeSeries(x, priorDays):
       new_x = []
       # 遍历每个元素的位置索引
       for i in range(len(x)):
           # 根据位置确定用于归一化的窗口
           if i >= 2 * priorDays:
               subarray = x.iloc[i - 2*priorDays : i - priorDays]
           else:
               # 数据长度不足时取现有全部数据
               subarray = x.iloc[0:priorDays] if len(x)>=priorDays else x
           # 避免除以0，取最大值和0.001的较大值
           max_value = max(subarray.max(), 0.001)
           new_x.append(x.iloc[i] / max_value)
       return new_x
   

3. 避免修改原数据的警告：在creatsequence函数开头复制一份数据，防止触发SettingWithCopyWarning：

   def creatsequence(data, length):
       data = data.copy()  # 新增该行，独立处理当前数据
       x = []
       y = []
       # 后续代码不变
   

内容的提问来源于stack exchange，提问作者Dmayall