使用numpy.corrcoef进行无监督特征选择时的索引错误排查

问题描述 #

需求为执行成对比较，筛选出DataFrame对象subset中相关性>0.05的行，但运行代码时触发IndexingError: Too many indexers。

原实现代码 #

import pandas as pd
import numpy as np

# 成对相关性计算
c = np.corrcoef(subset.T)
c = pd.DataFrame(c)
s = c.unstack()
so = s.sort_values(kind="quicksort", ascending=False)
so = np.abs(so)
#so = int(np.argmax(so))

# 筛选第三列值>0.05的结果
thresh2 = 0.05
so_sub = so.loc[so.iloc[:,-1:] > 0.05]
subset = subset.iloc[so_sub]

报错信息 #

---------------------------------------------------------------------------
IndexingError                             Traceback (most recent call last)
Input In [23], in <cell line: 23>()
     21 # Retrieve only if value in 3rd column > 0.05
     22 thresh2 = 0.05
---> 23 so_sub = so.loc[so.iloc[:,-1:] > 0.05]
     24 mrna_subset = mrna_subset.iloc[so_sub]

File ~/.local/lib/python3.9/site-packages/pandas/core/indexing.py:1097, in _LocationIndexer.__getitem__(self, key)
   1095     if self._is_scalar_access(key):
   1096         return self.obj._get_value(*key, takeable=self._takeable)
-> 1097     return self._getitem_tuple(key)
   1098 else:
   1099     # we by definition only have the 0th axis
   1100     axis = self.axis or 0

File ~/.local/lib/python3.9/site-packages/pandas/core/indexing.py:1594, in _iLocIndexer._getitem_tuple(self, tup)
   1593 def _getitem_tuple(self, tup: tuple):
-> 1594     tup = self._validate_tuple_indexer(tup)
   1595     with suppress(IndexingError):
   1596         return self._getitem_lowerdim(tup)

File ~/.local/lib/python3.9/site-packages/pandas/core/indexing.py:900, in _LocationIndexer._validate_tuple_indexer(self, key)
    895 @final
    896 def _validate_tuple_indexer(self, key: tuple) -> tuple:
    897     """
    898     Check the key for valid keys across my indexer.
    899     """
-> 900     key = self._validate_key_length(key)
    901     key = self._expand_ellipsis(key)
    902     for i, k in enumerate(key):

File ~/.local/lib/python3.9/site-packages/pandas/core/indexing.py:939, in _LocationIndexer._validate_key_length(self, key)
    937             raise IndexingError(_one_ellipsis_message)
    938         return self._validate_key_length(key)
-> 939     raise IndexingError("Too many indexers")
    940 return key

IndexingError: Too many indexers

输入数据示例 #

pd.DataFrame({'A2M': {'TCGA.2K.A9WE.01': 40686.22,
  'TCGA.2Z.A9J1.01': 11009.03,
  'TCGA.2Z.A9J3.01': 3180.79,
  'TCGA.2Z.A9J5.01': 16771.52},
 'A4GALT': {'TCGA.2K.A9WE.01': 2583.0,
  'TCGA.2Z.A9J1.01': 4720.0,
  'TCGA.2Z.A9J3.01': 2768.0,
  'TCGA.2Z.A9J5.01': 1689.0},
 'AAK1': {'TCGA.2K.A9WE.01': 2590.0,
  'TCGA.2Z.A9J1.01': 2562.0,
  'TCGA.2Z.A9J3.01': 2715.0,
  'TCGA.2Z.A9J5.01': 3010.0},
 'AAMP': {'TCGA.2K.A9WE.01': 4478.0,
  'TCGA.2Z.A9J1.01': 4518.0,
  'TCGA.2Z.A9J3.01': 5936.0,
  'TCGA.2Z.A9J5.01': 4139.0}})

报错原因 #

1. 索引维度错误：so是unstack()生成的多级索引Series，仅为一维结构，不存在“列”的概念。原代码中so.iloc[:,-1:]试图取“最后一列”，会返回一个带多级索引的Series切片，而非布尔筛选条件。
2. 索引器不匹配：将上述错误的切片传入so.loc[]，导致pandas识别到过多的索引维度，触发Too many indexers错误。
3. 逻辑不符需求：原代码计算的是特征列之间的相关性（subset.T转置后计算），但最后试图用相关性对的索引去筛选subset的行，逻辑不匹配，无法实现“筛选相关性>0.05的行”的需求。

修正方案 #

根据需求的两种常见场景，分别给出实现代码：

场景1：筛选样本行之间相关性绝对值>0.05的行 #

即保留所有与至少一个其他样本行相关性达标的行：

import pandas as pd
import numpy as np

# 计算样本行之间的成对相关性（每行对应一个样本）
corr_matrix = np.corrcoef(subset)
corr_df = pd.DataFrame(corr_matrix, index=subset.index, columns=subset.index)

# 展开成对关系，排除自相关（对角线值为1的元素）
corr_unstacked = corr_df.unstack()
corr_unstacked = corr_unstacked[corr_unstacked.index.get_level_values(0) != corr_unstacked.index.get_level_values(1)]

# 筛选相关性绝对值>0.05的成对关系
thresh = 0.05
corr_filtered = corr_unstacked[np.abs(corr_unstacked) > thresh]

# 获取所有参与达标相关性对的行索引（去重）
valid_rows = pd.Index(np.unique(corr_filtered.index.get_level_values(0)))

# 最终筛选目标行
subset_filtered = subset.loc[valid_rows]

场景2：筛选特征列之间相关性绝对值>0.05的列（保留对应列的所有行） #

若实际需求是保留相关性达标的特征列，而非行：

import pandas as pd
import numpy as np

# 计算特征列之间的成对相关性（每列对应一个特征）
corr_matrix = np.corrcoef(subset.T)
corr_df = pd.DataFrame(corr_matrix, index=subset.columns, columns=subset.columns)

# 展开成对关系，排除自相关和重复对（如(A,B)与(B,A)）
corr_unstacked = corr_df.unstack()
corr_unstacked = corr_unstacked[corr_unstacked.index.get_level_values(0) < corr_unstacked.index.get_level_values(1)]

# 筛选相关性绝对值>0.05的成对关系
thresh = 0.05
corr_filtered = corr_unstacked[np.abs(corr_unstacked) > thresh]

# 获取所有参与达标相关性对的特征列（去重）
valid_cols = pd.Index(np.unique(corr_filtered.index.get_level_values(0))).union(
    pd.Index(np.unique(corr_filtered.index.get_level_values(1)))
)

# 最终筛选目标列（保留对应列的所有行）
subset_filtered = subset[valid_cols]

内容的提问来源于stack exchange，提问作者Anon