能否为C# AutoML定义自定义优化指标？替代方案探讨

一、AutoML的限制 #

ML.NET AutoML的RegressionExperimentSettings.OptimizingMetric仅支持内置RegressionMetric枚举，无法直接传入自定义指标作为优化目标。仅能事后对AutoML生成的模型用自定义指标评估选优，但无法让AutoML在调参过程中以此指标为导向。

二、常规ML.NET实现步骤 #

1. 扩展数据模型 #

将查找表的四个区间指标列加入ModelInput，确保每个样本自带对应区间的指标值：

public class ModelInput
{
    [LoadColumn(0)]
    public float Label { get; set; }
    
    [LoadColumn(1, numberOfFeatures)]
    [VectorType(numberOfFeatures)]
    public float[] Features { get; set; }
    
    // 新增查找表对应区间的指标列
    [LoadColumn(numberOfFeatures + 1)]
    public float Metric0To25 { get; set; }
    
    [LoadColumn(numberOfFeatures + 2)]
    public float Metric25To50 { get; set; }
    
    [LoadColumn(numberOfFeatures + 3)]
    public float Metric50To75 { get; set; }
    
    [LoadColumn(numberOfFeatures + 4)]
    public float Metric75To100 { get; set; }
}

加载数据时需将原数据集与查找表指标合并为包含上述列的输入文件。

2. 自定义评估指标函数 #

实现函数计算数据集的自定义指标（即预测值对应区间的指标均值）：

public static float CalculateCustomMetric(IDataView dataView, MLContext mlContext)
{
    var predictions = mlContext.Data.CreateEnumerable<(float Score, float Metric0To25, float Metric25To50, float Metric50To75, float Metric75To100)>(dataView, reuseRowObject: false);
    
    float total = 0;
    int count = 0;
    
    foreach (var pred in predictions)
    {
        float metricValue;
        if (pred.Score is >= 0 and < 0.25)
            metricValue = pred.Metric0To25;
        else if (pred.Score is >= 0.25 and < 0.5)
            metricValue = pred.Metric25To50;
        else if (pred.Score is >= 0.5 and < 0.75)
            metricValue = pred.Metric50To75;
        else if (pred.Score is >= 0.75 and <= 1)
            metricValue = pred.Metric75To100;
        else
            metricValue = 0; // 处理超出0-1范围的预测值，可按需调整
        
        total += metricValue;
        count++;
    }
    
    return count == 0 ? 0 : total / count;
}

3. 手动枚举模型与参数实现AutoML式调参 #

手动枚举候选模型及参数组合，通过交叉验证评估自定义指标，筛选最优模型：

MLContext mlContext = new MLContext(42);
IDataView data = mlContext.Data.LoadFromTextFile<ModelInput>(dataPath.ToString(), separatorChar: ',', hasHeader: true);

// 定义候选模型与参数组合
var candidateModels = new List<(string Name, IEstimator<ITransformer> Estimator)>
{
    // SDCA回归器，不同L2正则化参数
    ("SDCA_L2_0.01", mlContext.Regression.Trainers.Sdca(new SdcaRegressionTrainer.Options 
        { LabelColumnName = "Label", FeatureColumnName = "Features", L2Regularization = 0.01f })),
    ("SDCA_L2_0.1", mlContext.Regression.Trainers.Sdca(new SdcaRegressionTrainer.Options 
        { LabelColumnName = "Label", FeatureColumnName = "Features", L2Regularization = 0.1f })),
    // LightGBM回归器，不同学习率
    ("LightGBM_LR_0.05", mlContext.Regression.Trainers.LightGbm(new LightGbmRegressionTrainer.Options 
        { LabelColumnName = "Label", FeatureColumnName = "Features", LearningRate = 0.05f })),
    ("LightGBM_LR_0.1", mlContext.Regression.Trainers.LightGbm(new LightGbmRegressionTrainer.Options 
        { LabelColumnName = "Label", FeatureColumnName = "Features", LearningRate = 0.1f }))
};

// 跟踪最优模型
float bestMetric = float.MinValue;
ITransformer bestModel = null;
string bestModelName = "";

// 遍历候选模型，执行10折交叉验证
foreach (var (modelName, estimator) in candidateModels)
{
    var cvResults = mlContext.Regression.CrossValidate(data, estimator, 10, "Label");
    
    // 计算交叉验证的自定义指标均值
    float avgMetric = cvResults.Sum(fold => CalculateCustomMetric(fold.Predicted, mlContext)) / cvResults.Count;
    
    Console.WriteLine($"模型 {modelName} 自定义指标均值：{avgMetric:F4}");
    
    if (avgMetric > bestMetric)
    {
        bestMetric = avgMetric;
        bestModel = cvResults.First().Model;
        bestModelName = modelName;
    }
}

// 用全量数据训练最优模型
var finalPipeline = candidateModels.First(m => m.Name == bestModelName).Estimator;
ITransformer finalBestModel = finalPipeline.Fit(data);

// 保存模型
mlContext.Model.Save(finalBestModel, data.Schema, "best_custom_model.zip");

4. 自定义损失函数（可选） #

若需训练过程直接以自定义指标为导向，可扩展ML.NET训练器实现自定义损失（如基于SDCA或TensorFlow），但实现复杂度较高，多数场景下上述手动调参方案已足够。

内容的提问来源于stack exchange，提问作者automl3009