Python AIF360库OptimPreproc超大内存分配错误求助

使用Python的AIF360库时，触发内存错误：

MemoryError: Unable to allocate 42.7 TiB for an array with shape (46926397440000,) and data type int8

数据集最多仅1000行，其他预处理技术（如LFR、Reweighing等）均可正常运行，说明数据集结构和指标无问题。以下是复现代码，恳请有OptimPreproc使用经验的人士提供解决建议：

复现代码（基于德国信用数据集） #

依赖库安装与导入 #

!pip install aif360
from aif360.algorithms.preprocessing import LFR
import pandas as pd
import numpy as np
import scipy.optimize as optim

from aif360.algorithms import Transformer as TR
from aif360.algorithms.preprocessing.lfr_helpers import helpers as lfr_helpers
from aif360.algorithms.preprocessing import Reweighing

import pandas as pd
import numpy as np

import matplotlib
import matplotlib.pyplot as plt
import matplotlib.patches as patches

from aif360.datasets import StandardDataset
from aif360.metrics import BinaryLabelDatasetMetric, ClassificationMetric
from aif360.algorithms.preprocessing import LFR, Reweighing

from sklearn.metrics import confusion_matrix, accuracy_score, f1_score, roc_curve, auc
from sklearn.preprocessing import MinMaxScaler, LabelEncoder
from sklearn.ensemble import RandomForestClassifier
#import tensorflow as tf

from aif360.datasets import GermanDataset, BinaryLabelDataset
from aif360.metrics import BinaryLabelDatasetMetric
from aif360.algorithms.preprocessing import DisparateImpactRemover
from aif360.algorithms.preprocessing import OptimPreproc

from IPython.display import Markdown, display

import cvxpy as cp
import numpy as np
from aif360.algorithms.preprocessing.optim_preproc_helpers.opt_tools import OptTools

数据加载与预处理 #

data = pd.read_csv('original_data.csv')
data.head() 

性别维度数据集处理 #

# 性别维度示例
dataset_gender = StandardDataset(data, 
                          label_name='Creditability', 
                          favorable_classes=[1], 
                          protected_attribute_names=['Gender'], 
                          privileged_classes=[[0]],
                          features_to_drop=['Age', 'Foreign_worker', 'Age_grouped'])
        
dataset_orig_train_gender, dataset_orig_test_gender = dataset_gender.split([0.7], shuffle=True)

privileged_groups_gender = [{'Gender': 0}]
unprivileged_groups_gender = [{'Gender': 1}]

metric_orig_train_gender = BinaryLabelDatasetMetric(dataset_orig_train_gender, 
                                             unprivileged_groups=unprivileged_groups_gender,
                                             privileged_groups=privileged_groups_gender)

print("SPD gender= %f" % metric_orig_train_gender.statistical_parity_difference())
print("DI gender = %f" % metric_orig_train_gender.disparate_impact())

触发错误的优化预处理代码 #

optim_options = {
    "epsilon": 0.05,
    "clist": [0.99, 1.99, 2.99],
    "dlist": [.1, 0.05, 0]
}
    
OP = OptimPreproc(OptTools, optim_options)
OP = OP.fit(dataset_orig_train_gender)

dataset_preprocess_train_gender = OP.transform(dataset_orig_train_gender, transform_Y=True)
dataset_preprocess_train_gender = dataset_orig_train_gender.align_datasets(dataset_preprocess_train_gender)

可能的解决建议 #

• 优化参数配置：clist和dlist的组合会生成多组优化任务，原参数3*3=9组，若特征数量较多，会导致中间计算的数组规模爆炸。建议先缩减参数数量，比如clist=[1.99]、dlist=[0.05]，测试是否还触发内存错误。
• 减少特征数量：即使行数少，高维特征会大幅增加OptimPreproc的计算复杂度。先保留核心特征（比如5-10个）运行代码，验证是否是特征维度导致的问题。
• 检查AIF360版本：部分版本的OptimPreproc存在内存泄漏或计算逻辑bug，尝试切换到稳定版本（如0.6.0或0.7.0）。
• 调整transform参数：先设置transform_Y=False运行transform，排除标签转换带来的内存压力，再逐步排查问题点。
• 验证OptTools初始化：确认OptTools是否需要额外参数，部分场景下可能需要指定约束类型或优化器参数，避免默认逻辑生成超大数组。

内容的提问来源于stack exchange，提问作者FLKYZ