SKLearn Isolation Forest模型持续训练复用及特征维度匹配问题求解

一、核心问题根源 #

你遇到的特征维度不匹配，本质是新数据的One-Hot编码特征集合与训练时不一致——要么新数据出现了训练时没见过的分类值，生成了新特征列；要么缺失了部分训练时存在的特征列。手动用set.union处理仅适用于首次划分的训练/测试集，无法适配后续动态新增的数据。

二、推理阶段特征对齐解决方案 #

推荐用Sklearn官方的预处理组件替代手动列处理，从根源保证特征一致性：

方案1：使用Pipeline整合预处理与模型（最推荐） #

将One-Hot编码、模型训练打包成一个Pipeline，保存整个管道后，推理时直接用管道处理新数据，自动对齐特征。

训练阶段代码

import pickle
import pandas as pd
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline

# 定义分类/数值特征列
categorical_cols = ["分类列1", "分类列2"]
numerical_cols = ["数值列1", "数值列2"]

# 构建预处理规则：分类特征做One-Hot，数值特征直接保留
preprocessor = ColumnTransformer(
    transformers=[
        ('cat', OneHotEncoder(sparse_output=False, handle_unknown='ignore'), categorical_cols),
        ('num', 'passthrough', numerical_cols)
    ])

# 整合预处理与模型成管道
model_pipeline = Pipeline([
    ('preprocess', preprocessor),
    ('model', IsolationForest())
])

# 训练并保存管道
model_pipeline.fit(training_data)
with open('isolation_forest_pipeline.pkl', 'wb') as f:
    pickle.dump(model_pipeline, f)

• handle_unknown='ignore'：遇到训练时未见过的分类值，不会生成新特征列，避免维度膨胀。

CRON推理阶段代码

import pickle
import pandas as pd

# 加载保存的管道
with open('isolation_forest_pipeline.pkl', 'rb') as f:
    model_pipeline = pickle.load(f)

# 加载新数据并直接预测
new_data = pd.read_csv('new_data.csv')
predictions = model_pipeline.predict(new_data)
scores = model_pipeline.score_samples(new_data)

方案2：单独保存编码器与模型（兼容现有代码） #

如果不想用Pipeline，可以单独保存训练时的编码器，推理时用同一编码器处理新数据：

训练阶段代码

import pickle
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer

# 训练编码器
preprocessor = ColumnTransformer(
    transformers=[
        ('cat', OneHotEncoder(sparse_output=False, handle_unknown='ignore'), categorical_cols),
        ('num', 'passthrough', numerical_cols)
    ])
preprocessor.fit(training_data)

# 训练并保存模型
processed_train = preprocessor.transform(training_data)
model = IsolationForest()
model.fit(processed_train)

# 保存编码器与模型
with open('preprocessor.pkl', 'wb') as f:
    pickle.dump(preprocessor, f)
with open('isolation_forest_model.pkl', 'wb') as f:
    pickle.dump(model, f)

CRON推理阶段代码

import pickle
import pandas as pd

# 加载编码器与模型
with open('preprocessor.pkl', 'rb') as f:
    preprocessor = pickle.load(f)
with open('isolation_forest_model.pkl', 'rb') as f:
    model = pickle.load(f)

# 预处理新数据并预测
new_data = pd.read_csv('new_data.csv')
processed_new = preprocessor.transform(new_data)
predictions = model.predict(processed_new)
scores = model.score_samples(processed_new)

方案3：兼容手动get_dummies的特征对齐（不推荐） #

如果坚持用pandas的get_dummies，需保存训练时的特征列名，推理时强制对齐：

训练阶段代码

import pickle
import pandas as pd

# 对训练数据做One-Hot编码
encoded_train = pd.get_dummies(training_data, columns=categorical_cols)
# 保存训练时的特征列名
with open('train_feature_cols.pkl', 'wb') as f:
    pickle.dump(list(encoded_train.columns), f)

# 训练并保存模型
model = IsolationForest()
model.fit(encoded_train)
with open('isolation_forest_model.pkl', 'wb') as f:
    pickle.dump(model, f)

CRON推理阶段代码

import pickle
import pandas as pd

# 加载特征列名与模型
with open('train_feature_cols.pkl', 'rb') as f:
    train_cols = pickle.load(f)
model = pickle.load(open('isolation_forest_model.pkl', 'rb'))

# 新数据编码并对齐特征
encoded_new = pd.get_dummies(new_data, columns=categorical_cols)
encoded_new_aligned = encoded_new.reindex(columns=train_cols, fill_value=0)

# 预测
predictions = model.predict(encoded_new_aligned)
scores = model.score_samples(encoded_new_aligned)

三、CRON调度下的持续训练方案 #

由于Sklearn的IsolationForest不支持增量训练（partial_fit），需采用全量重训的方式更新模型：

定时训练脚本示例 #

import pickle
import pandas as pd
from sklearn.pipeline import Pipeline

# 加载历史数据（比如保留最近30天的数据，避免数据量过大）
historical_data = pd.read_csv('recent_30d_data.csv')
# 加载本次新增数据
new_data = pd.read_csv('new_data.csv')
# 合并并去重
combined_data = pd.concat([historical_data, new_data]).drop_duplicates()

# 加载旧管道（首次训练则新建）
try:
    with open('isolation_forest_pipeline.pkl', 'rb') as f:
        model_pipeline = pickle.load(f)
except FileNotFoundError:
    # 首次训练构建管道（同方案1的训练代码）
    from sklearn.ensemble import IsolationForest
    from sklearn.preprocessing import OneHotEncoder
    from sklearn.compose import ColumnTransformer

    categorical_cols = ["分类列1", "分类列2"]
    numerical_cols = ["数值列1", "数值列2"]

    preprocessor = ColumnTransformer(
        transformers=[
            ('cat', OneHotEncoder(sparse_output=False, handle_unknown='ignore'), categorical_cols),
            ('num', 'passthrough', numerical_cols)
        ])

    model_pipeline = Pipeline([
        ('preprocess', preprocessor),
        ('model', IsolationForest())
    ])

# 用合并数据重训模型
model_pipeline.fit(combined_data)

# 保存更新后的管道与历史数据
with open('isolation_forest_pipeline.pkl', 'wb') as f:
    pickle.dump(model_pipeline, f)
combined_data.to_csv('recent_30d_data.csv', index=False)

# 对新数据执行预测
predictions = model_pipeline.predict(new_data)
scores = model_pipeline.score_samples(new_data)

关键注意事项 #

• 用滑动窗口数据训练（比如保留最近90天），避免数据量过大导致训练耗时过长
• 每次训练后覆盖旧模型文件，确保CRON脚本加载最新模型
• 定期验证模型性能（比如用验证集检测异常检测的召回率），避免模型退化
• 确保训练与推理时的Sklearn版本一致，避免pickle跨版本兼容性问题

内容的提问来源于stack exchange，提问作者MaJunior