CreateML训练的人脸情感CoreML模型在macOS与iPadOS表现差异问题

核心问题分析 #

模型在CreateML预览、Mac端正常，但iPad端98%识别为积极情感，核心原因大概率是图像输入的方向/格式不匹配，或是设备硬件导致的图像预处理差异，结合代码来看，主要有以下几个修复方向：

1. 修正相机方向与EXIF映射错误 #

前置摄像头采集的画面本身是镜像状态，CreateML预览会自动处理，但你的代码未区分前后置摄像头的方向映射，导致iPad上模型接收到的人脸是左右翻转的，和训练数据不匹配，进而误判。

修改exifOrientationFromDeviceOrientation方法：

public func exifOrientationFromDeviceOrientation() -> CGImagePropertyOrientation {
    let curDeviceOrientation = UIDevice.current.orientation
    let cameraPosition = (session.inputs.first as? AVCaptureDeviceInput)?.device.position
    let exifOrientation: CGImagePropertyOrientation
    
    switch curDeviceOrientation {
    case .portraitUpsideDown:
        exifOrientation = cameraPosition == .front ? .rightMirrored : .left
    case .landscapeLeft:
        exifOrientation = cameraPosition == .front ? .downMirrored : .upMirrored
    case .landscapeRight:
        exifOrientation = cameraPosition == .front ? .upMirrored : .down
    case .portrait:
        exifOrientation = cameraPosition == .front ? .leftMirrored : .up
    default:
        exifOrientation = cameraPosition == .front ? .leftMirrored : .up
    }
    return exifOrientation
}

2. 强制匹配摄像头分辨率与模型输入尺寸 #

iPad摄像头默认输出高分辨率画面，和CreateML训练时的输入尺寸（比如你注释的.cif352x288）不匹配，Vision自动缩放会导致人脸区域失真，引发误判。

取消注释并设置对应分辨率：

session.beginConfiguration()
session.sessionPreset = .cif352x288 // 确保和CreateML训练时的输入尺寸完全一致

3. 排查CoreML硬件加速差异 #

iPad和Mac的CoreML运行环境存在差异，默认配置可能使用了不同的执行后端，导致推理结果偏移。先强制用CPU推理测试：

func setupVision() {
    do {
        let config = MLModelConfiguration()
        config.computeUnits = .cpuOnly // 先排除硬件加速的影响
        let faceModel = try FaceEmotion(configuration: config)
        let visionModel = try VNCoreMLModel(for: faceModel.model)
        // 剩余代码不变
    } catch let error as NSError {
        print("Model loading went wrong: \(error)")
    }
}

若CPU推理正常，再尝试将computeUnits改回.all重新测试。

4. 添加人脸检测前置过滤 #

你的代码直接将整帧画面输入模型，但训练数据应该只针对人脸区域，iPad上可能因人脸占比小、背景干扰导致误判。添加人脸检测，只把人脸区域输入模型：

// 先添加属性保存最后一帧
private var lastPixelBuffer: CVPixelBuffer?

func setupVision() {
    do {
        let config = MLModelConfiguration()
        let faceModel = try FaceEmotion(configuration: config)
        let visionModel = try VNCoreMLModel(for: faceModel.model)
        
        // 先做人脸检测
        let faceDetectionRequest = VNDetectFaceRectanglesRequest { [weak self] request, error in
            guard let results = request.results as? [VNFaceObservation], let face = results.first else {
                DispatchQueue.main.async {
                    self?.sentiment = "No face detected"
                }
                return
            }
            
            // 仅对人脸区域执行情感检测
            let emotionRequest = VNCoreMLRequest(model: visionModel) { request, error in
                guard let results = request.results as? [VNClassificationObservation], let topResult = results.first else {
                    return
                }
                DispatchQueue.main.async {
                    self?.sentiment = "\(topResult.identifier) \(topResult.confidence)"
                }
            }
            emotionRequest.regionOfInterest = face.boundingBox
            
            // 执行请求
            if let pixelBuffer = self?.lastPixelBuffer {
                let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: self?.exifOrientationFromDeviceOrientation() ?? .up)
                try? handler.perform([emotionRequest])
            }
        }
        
        self.requests = [faceDetectionRequest]
    } catch let error as NSError {
        print("Model loading went wrong: \(error)")
    }
}

// 修改captureOutput保存当前帧
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
        return
    }
    lastPixelBuffer = pixelBuffer
    
    let exifOrientation = exifOrientationFromDeviceOrientation()
    let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: exifOrientation, options: [:])
    do {
        try imageRequestHandler.perform(self.requests)
    } catch {
        print(error)
    }
}

验证步骤 #

1. 优先修改方向映射代码，测试iPad识别结果
2. 若仍异常，强制设置摄像头分辨率匹配模型输入尺寸
3. 再尝试添加人脸检测前置过滤
4. 最后排查CoreML硬件加速差异

内容的提问来源于stack exchange，提问作者user1969