如何将CMSampleBuffer转为CVImageBuffer？MoodMe人脸识别适配问题

Hey there! I see you're having trouble getting MoodMe's face detection to work when passing a UIImage, and you're wondering how to use the CVImageBuffer-based method instead. Let's break this down and fix your code.

First: Yes, you can get CVImageBuffer from CMSampleBuffer #

Good news—you're already halfway there! The CMSampleBufferGetImageBuffer(sampleBuffer) call you're using directly returns a CVImageBuffer? (specifically a CVPixelBuffer, which is a subclass of CVImageBuffer). That's exactly what MoodMe's processImageBuffer(frame:) method expects.

Why UIImage isn't working #

Most likely, converting the camera's raw output to a UIImage introduces issues like incorrect color space conversion, orientation mismatches, or loss of image metadata that MoodMe relies on for face detection. The raw CVImageBuffer from the camera is the optimal input for the framework.

Modified Working Code #

Here's how to adjust your captureOutput method to use the CVImageBuffer directly, plus fix common orientation issues that might still block detection:

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    // Safely unwrap the pixel buffer (CVImageBuffer subclass)
    guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
        print("Failed to get pixel buffer from sample buffer")
        return
    }
    
    // Critical: Set correct video orientation to match device UI
    // This ensures MoodMe gets the face in the right orientation
    if let videoOutput = output as? AVCaptureVideoDataOutput, connection.isVideoOrientationSupported {
        let orientation: AVCaptureVideoOrientation
        switch UIDevice.current.orientation {
        case .portrait:
            orientation = .portrait
        case .landscapeLeft:
            orientation = .landscapeRight
        case .landscapeRight:
            orientation = .landscapeLeft
        case .portraitUpsideDown:
            orientation = .portraitUpsideDown
        default:
            orientation = .portrait
        }
        connection.videoOrientation = orientation
    }
    
    // Use the CVImageBuffer directly with MoodMe
    mdm.processImageBuffer(frame: pixelBuffer)
    
    // Check face tracking status
    if mdm.faceTracked {
        print("Face detected!")
    } else {
        print("No face found")
    }
}

Key Notes #

• Orientation Handling: Camera output defaults to landscape even if your app is in portrait. Setting the connection.videoOrientation ensures the frame is rotated correctly, which is often the hidden reason face detection fails even with the right buffer type.
• Safe Unwrapping: Always guard optional values like pixelBuffer to avoid unexpected crashes.
• Camera Session Preset: Make sure your AVCaptureSession uses a preset with sufficient resolution (e.g., AVCaptureSession.Preset.hd1280x720 or higher) — low-resolution frames can prevent the framework from picking up faces.
• Permissions: Double-check you've added NSCameraUsageDescription to your Info.plist and requested camera access from the user; missing permissions will silently break camera output.

内容的提问来源于stack exchange，提问作者Igor Custodio