ML.Net加载ONNX模型做内存图像分类时IDataView类型识别报错求助

错误原因 #

ML.NET无法直接识别System.Drawing.Bitmap类型作为IDataView的成员，因此抛出Could not determine an IDataView type...错误。ML.NET内置支持的图像输入类型为字节数组或ImageSharp的Image类型，而非Bitmap。

解决方案1：使用字节数组作为输入 #

1. 修改输入类 #

将Bitmap替换为byte[]，通过ColumnName指定对应的数据列名：

public class InputData
{
    [ColumnName("Image")]
    public byte[] ImageBytes { get; set; }
}

public class OutputData
{
    [ColumnName("dense")]
    public float[] Scores { get; set; }
}

2. 调整初始化代码 #

Pipeline无需大幅修改，ResizeImages和ExtractPixels可直接处理字节数组格式的图像：

public PredictionEngine<InputData, OutputData> Initialize(string path)
{
    MLContext mlContext = new MLContext();

    int size = 224;
    var pipeline = mlContext.Transforms.ResizeImages(
            outputColumnName: "image", 
            imageWidth: size, 
            imageHeight: size, 
            inputColumnName: "Image")
        .Append(mlContext.Transforms.ExtractPixels(outputColumnName: "image"))
        .Append(mlContext.Transforms.ApplyOnnxModel(
            outputColumnName: "dense", 
            inputColumnName: "image", 
            modelFile: path, 
            fallbackToCpu: true));

    var data = mlContext.Data.LoadFromEnumerable(Array.Empty<InputData>());
    var model = pipeline.Fit(data);

    return mlContext.Model.CreatePredictionEngine<InputData, OutputData>(model);
}

3. 预测时转换Bitmap为字节数组 #

将内存中的Bitmap转换为字节数组后传入预测引擎：

// 辅助方法：Bitmap转字节数组
private static byte[] ConvertBitmapToByteArray(Bitmap bitmap)
{
    using (var ms = new MemoryStream())
    {
        bitmap.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg);
        return ms.ToArray();
    }
}

// 预测示例
var input = new InputData
{
    ImageBytes = ConvertBitmapToByteArray(yourMemoryBitmap)
};
var prediction = predictionEngine.Predict(input);

解决方案2：使用ImageSharp的Image类型（推荐） #

ML.NET对ImageSharp有原生支持，无需手动转换字节数组，步骤如下：

1. 安装NuGet包 #

安装Microsoft.ML.ImageSharp和SixLabors.ImageSharp：

Install-Package Microsoft.ML.ImageSharp
Install-Package SixLabors.ImageSharp

2. 修改输入类 #

使用Image<RGB24>作为图像类型，保留ImageType属性：

using SixLabors.ImageSharp;
using SixLabors.ImageSharp.PixelFormats;

public class InputData
{
    [ColumnName("Image")]
    [ImageType(720, 1280)]
    public Image<RGB24> Image { get; set; }
}

public class OutputData
{
    [ColumnName("dense")]
    public float[] Scores { get; set; }
}

3. 初始化代码保持不变 #

原Pipeline代码可直接使用，ML.NET会自动处理ImageSharp类型：

public PredictionEngine<InputData, OutputData> Initialize(string path)
{
    MLContext mlContext = new MLContext();

    int size = 224;
    var pipeline = mlContext.Transforms.ResizeImages(
            outputColumnName: "image", 
            imageWidth: size, 
            imageHeight: size, 
            inputColumnName: "Image")
        .Append(mlContext.Transforms.ExtractPixels(outputColumnName: "image"))
        .Append(mlContext.Transforms.ApplyOnnxModel(
            outputColumnName: "dense", 
            inputColumnName: "image", 
            modelFile: path, 
            fallbackToCpu: true));

    var data = mlContext.Data.LoadFromEnumerable(Array.Empty<InputData>());
    var model = pipeline.Fit(data);

    return mlContext.Model.CreatePredictionEngine<InputData, OutputData>(model);
}

4. 预测时转换Bitmap为Image #

将Bitmap转换为ImageSharp的Image类型：

// 辅助方法：Bitmap转Image<RGB24>
private static Image<RGB24> ConvertBitmapToImageSharp(Bitmap bitmap)
{
    using (var ms = new MemoryStream())
    {
        bitmap.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg);
        ms.Position = 0;
        return Image.Load<RGB24>(ms);
    }
}

// 预测示例
var input = new InputData
{
    Image = ConvertBitmapToImageSharp(yourMemoryBitmap)
};
var prediction = predictionEngine.Predict(input);

内容的提问来源于stack exchange，提问作者pascx64