如何高效遍历特征组合并基于MAE排序LightGBM回归模型结果

首先明确：30个特征的所有非空组合有2^30-1≈10亿种，暴力逐个计算完全不现实，必须用启发式/基于模型的特征选择方法来高效缩小搜索范围，找到近似最优的特征组合。

以下是几种可行的高效实现方案：

一、贪心式特征选择（向前/向后） #

这是最易实现的高效方法，通过逐步添加/移除特征来逼近最优组合：

向前选择（从空集开始逐步添加最优特征） #

import lightgbm as lgb
import pandas as pd
from sklearn.metrics import mean_absolute_error

# 初始化数据
target = 'Target'
features = [col for col in train.columns if col != target]
X_train, y_train = train[features], train[target]
X_test, y_test = test[features], test[target]

selected_features = []
mae_history = []
best_mae = float('inf')

while len(selected_features) < len(features):
    # 遍历未选中的特征，计算添加后的MAE
    candidate_mae = {}
    for feat in set(features) - set(selected_features):
        current_features = selected_features + [feat]
        # 训练LightGBM（可加入早停提升效率）
        reg = lgb.LGBMRegressor(random_state=42)
        reg.fit(X_train[current_features], y_train,
                eval_set=[(X_test[current_features], y_test)],
                early_stopping_rounds=10,
                eval_metric='mae',
                verbose=False)
        pred = reg.predict(X_test[current_features], num_iteration=reg.best_iteration_)
        mae = mean_absolute_error(y_test, pred)
        candidate_mae[feat] = mae
    
    # 选MAE最小的特征加入
    best_feat = min(candidate_mae, key=candidate_mae.get)
    best_current_mae = candidate_mae[best_feat]
    selected_features.append(best_feat)
    mae_history.append((', '.join(selected_features), best_current_mae))
    
    # 如果MAE不再下降，提前终止
    if best_current_mae >= best_mae:
        break
    best_mae = best_current_mae

# 整理结果并排序
result_df = pd.DataFrame(mae_history, columns=['Features_used', 'MAE'])
result_df['Rank'] = result_df['MAE'].rank(method='min').astype(int)
result_df = result_df.sort_values('Rank').reset_index(drop=True)
print(result_df)

向后选择（从全特征开始逐步移除最差特征） #

逻辑类似，只是从全特征出发，每次移除使MAE上升最少的特征，适合需要精简特征数量的场景。

二、基于特征重要性的筛选 #

利用LightGBM自带的特征重要性，先筛选高重要性特征，再验证不同数量的特征组合：

import lightgbm as lgb
import pandas as pd
from sklearn.metrics import mean_absolute_error

target = 'Target'
features = [col for col in train.columns if col != target]
X_train, y_train = train[features], train[target]
X_test, y_test = test[features], test[target]

# 训练全特征模型，获取特征重要性
reg = lgb.LGBMRegressor(random_state=42)
reg.fit(X_train, y_train)
feat_importance = pd.Series(reg.feature_importances_, index=features).sort_values(ascending=False)

# 遍历不同数量的Top特征，计算MAE
mae_results = []
for k in range(1, len(features)+1):
    top_features = feat_importance.index[:k].tolist()
    reg_k = lgb.LGBMRegressor(random_state=42)
    reg_k.fit(X_train[top_features], y_train,
              eval_set=[(X_test[top_features], y_test)],
              early_stopping_rounds=10,
              eval_metric='mae',
              verbose=False)
    pred = reg_k.predict(X_test[top_features], num_iteration=reg_k.best_iteration_)
    mae = mean_absolute_error(y_test, pred)
    mae_results.append((', '.join(top_features), mae))

# 整理排序结果
result_df = pd.DataFrame(mae_results, columns=['Features_used', 'MAE'])
result_df['Rank'] = result_df['MAE'].rank(method='min').astype(int)
result_df = result_df.sort_values('Rank').reset_index(drop=True)
print(result_df)

三、启发式搜索算法（遗传算法） #

对于需要捕捉复杂特征交互的场景，可使用遗传算法这类启发式方法，通过模拟进化过程高效搜索最优组合，推荐用sklearn-genetic-opt库实现：

from sklearn_genetic import GASearchCV
from sklearn_genetic.space import Categorical
from sklearn.pipeline import Pipeline
import lightgbm as lgb
import pandas as pd
from sklearn.metrics import mean_absolute_error

target = 'Target'
features = [col for col in train.columns if col != target]
X_train, y_train = train[features], train[target]
X_test, y_test = test[features], test[target]

# 定义特征选择的搜索空间：每个特征可选或不选
def generate_feature_combinations(features):
    # 生成部分代表性组合（避免空间过大）
    combinations = []
    # 加入单个特征、两两组合、全特征等
    for i in range(len(features)):
        combinations.append([features[i]])
        for j in range(i+1, len(features)):
            combinations.append([features[i], features[j]])
    combinations.append(features)
    return combinations

search_space = {
    'regressor__feature_name': Categorical(generate_feature_combinations(features))
}

# 定义模型管道
pipe = Pipeline([
    ('regressor', lgb.LGBMRegressor(random_state=42))
])

# 遗传算法搜索
ga = GASearchCV(
    estimator=pipe,
    cv=3,
    scoring='neg_mean_absolute_error',
    population_size=20,
    generations=10,
    verbose=True,
    random_state=42
)

ga.fit(X_train, y_train)

# 提取最优组合并在测试集验证
best_features = ga.best_params_['regressor__feature_name']
reg_best = lgb.LGBMRegressor(random_state=42)
reg_best.fit(X_train[best_features], y_train)
best_mae = mean_absolute_error(y_test, reg_best.predict(X_test[best_features]))

# 整理结果（可扩展收集多代优秀组合）
result_df = pd.DataFrame([
    {'Features_used': ', '.join(best_features), 'MAE': best_mae, 'Rank': 1}
])
print(result_df)

关键优化点 #

• 所有方法均为近似最优，因为暴力遍历所有组合完全不可行；
• 训练LightGBM时加入early_stopping_rounds可大幅减少训练时间；
• 固定random_state保证结果可复现；
• 若数据集规模大，可先做特征预筛选（如方差过滤）进一步缩小范围。

内容的提问来源于stack exchange，提问作者Datalearner