基于AVCaptureSession：如何将CMSampleBuffer从BGRA转YUV 420f？

Great question—converting between pixel formats after OpenCV processing is a common need in iOS/macOS video projects. Since you've already set up your AVCaptureVideoDataOutput to output BGRA frames, here are two robust, production-ready approaches to convert those CMSampleBuffers to YUV 420f (the bi-planar variants Apple’s video stack uses):

Approach 1: CoreImage (Simpler, Good for Prototyping) #

CoreImage offers a straightforward way to render frames into different pixel formats, leveraging hardware acceleration by default. It’s easy to implement, though it might have slightly higher overhead than VideoToolbox for real-time high-resolution video.

Implementation Code #

- (void)captureOutput:(AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
    // Extract the BGRA pixel buffer from the input sample buffer
    CVPixelBufferRef sourceBGRA = CMSampleBufferGetImageBuffer(sampleBuffer);
    if (!sourceBGRA) return;

    // Create a CIImage from the BGRA buffer
    CIImage *processedImage = [CIImage imageWithCVPixelBuffer:sourceBGRA];
    // ⚠️ Insert your OpenCV processing here if you haven't already

    // Define settings for the YUV 420f destination buffer
    size_t frameWidth = CVPixelBufferGetWidth(sourceBGRA);
    size_t frameHeight = CVPixelBufferGetHeight(sourceBGRA);
    NSDictionary *destBufferAttrs = @{
        (id)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange),
        (id)kCVPixelBufferWidthKey: @(frameWidth),
        (id)kCVPixelBufferHeightKey: @(frameHeight),
        (id)kCVPixelBufferIOSurfacePropertiesKey: @{} // Enable hardware-backed buffer
    };

    // Create the destination YUV buffer
    CVPixelBufferRef destYUV = NULL;
    CVReturn cvStatus = CVPixelBufferCreate(kCFAllocatorDefault,
                                            frameWidth,
                                            frameHeight,
                                            kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange,
                                            (__bridge CFDictionaryRef)destBufferAttrs,
                                            &destYUV);
    if (cvStatus != kCVReturnSuccess || !destYUV) {
        NSLog(@"Failed to create YUV destination buffer");
        return;
    }

    // Render the processed image into the YUV buffer using hardware acceleration
    CIContext *ciContext = [CIContext contextWithOptions:@{kCIContextUseSoftwareRenderer: @NO}];
    [ciContext render:processedImage toCVPixelBuffer:destYUV];

    // Package the YUV buffer into a CMSampleBuffer for display
    CMTime presentationTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
    CMTime frameDuration = CMSampleBufferGetDuration(sampleBuffer);

    CMVideoFormatDescriptionRef videoFormat = NULL;
    cvStatus = CMVideoFormatDescriptionCreateForImageBuffer(NULL, destYUV, &videoFormat);
    if (cvStatus != kCVReturnSuccess || !videoFormat) {
        NSLog(@"Failed to create video format description");
        CVPixelBufferRelease(destYUV);
        return;
    }

    CMSampleTimingInfo timingInfo = {frameDuration, presentationTime, kCMTimeInvalid};
    CMSampleBufferRef outputSampleBuffer = NULL;
    cvStatus = CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault,
                                                  destYUV,
                                                  true,
                                                  NULL,
                                                  NULL,
                                                  videoFormat,
                                                  &timingInfo,
                                                  &outputSampleBuffer);
    if (cvStatus != kCVReturnSuccess || !outputSampleBuffer) {
        NSLog(@"Failed to create output sample buffer");
        CMVideoFormatDescriptionRelease(videoFormat);
        CVPixelBufferRelease(destYUV);
        return;
    }

    // Use outputSampleBuffer for display (e.g., feed to AVPlayerLayer or custom renderer)
    // ... your display logic here ...

    // Clean up resources to avoid leaks
    CMSampleBufferRelease(outputSampleBuffer);
    CMVideoFormatDescriptionRelease(videoFormat);
    CVPixelBufferRelease(destYUV);
}

Approach 2: VideoToolbox (High Performance, Real-Time Ready) #

VideoToolbox’s VTPixelTransferSession is built for hardware-accelerated pixel format conversion, making it perfect for real-time video processing where low latency and high throughput matter most.

Implementation Code #

First, initialize the transfer session (do this once during setup):

@property (nonatomic, assign) VTPixelTransferSessionRef pixelTransferSession;

- (void)setupPixelTransferSession {
    if (_pixelTransferSession) return;
    CVReturn cvStatus = VTPixelTransferSessionCreate(kCFAllocatorDefault, &_pixelTransferSession);
    if (cvStatus != kCVReturnSuccess) {
        NSLog(@"Failed to create VTPixelTransferSession");
    }
}

Then handle the conversion in your capture callback:

- (void)captureOutput:(AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
    CVPixelBufferRef sourceBGRA = CMSampleBufferGetImageBuffer(sampleBuffer);
    if (!sourceBGRA || !_pixelTransferSession) return;

    // ⚠️ Insert your OpenCV processing here
    // ...

    size_t frameWidth = CVPixelBufferGetWidth(sourceBGRA);
    size_t frameHeight = CVPixelBufferGetHeight(sourceBGRA);

    // Create destination YUV buffer
    NSDictionary *destBufferAttrs = @{
        (id)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange),
        (id)kCVPixelBufferWidthKey: @(frameWidth),
        (id)kCVPixelBufferHeightKey: @(frameHeight),
        (id)kCVPixelBufferIOSurfacePropertiesKey: @{}
    };

    CVPixelBufferRef destYUV = NULL;
    CVReturn cvStatus = CVPixelBufferCreate(kCFAllocatorDefault,
                                            frameWidth,
                                            frameHeight,
                                            kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange,
                                            (__bridge CFDictionaryRef)destBufferAttrs,
                                            &destYUV);
    if (cvStatus != kCVReturnSuccess || !destYUV) {
        NSLog(@"Failed to create YUV destination buffer");
        return;
    }

    // Configure transfer session for BGRA → YUV conversion
    VTPixelTransferSessionSetProperty(_pixelTransferSession,
                                      kVTPixelTransferPropertyKey_SourcePixelFormat,
                                      @(kCVPixelFormatType_32BGRA));
    VTPixelTransferSessionSetProperty(_pixelTransferSession,
                                      kVTPixelTransferPropertyKey_DestinationPixelFormat,
                                      @(kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange));

    // Execute the pixel transfer (hardware-accelerated)
    cvStatus = VTPixelTransferSessionTransferImage(_pixelTransferSession, sourceBGRA, destYUV);
    if (cvStatus != kCVReturnSuccess) {
        NSLog(@"Pixel transfer failed with error: %d", (int)cvStatus);
        CVPixelBufferRelease(destYUV);
        return;
    }

    // Package into CMSampleBuffer (same as CoreImage approach)
    CMTime presentationTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
    CMTime frameDuration = CMSampleBufferGetDuration(sampleBuffer);

    CMVideoFormatDescriptionRef videoFormat = NULL;
    cvStatus = CMVideoFormatDescriptionCreateForImageBuffer(NULL, destYUV, &videoFormat);
    if (cvStatus != kCVReturnSuccess || !videoFormat) {
        NSLog(@"Failed to create video format description");
        CVPixelBufferRelease(destYUV);
        return;
    }

    CMSampleTimingInfo timingInfo = {frameDuration, presentationTime, kCMTimeInvalid};
    CMSampleBufferRef outputSampleBuffer = NULL;
    cvStatus = CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault,
                                                  destYUV,
                                                  true,
                                                  NULL,
                                                  NULL,
                                                  videoFormat,
                                                  &timingInfo,
                                                  &outputSampleBuffer);
    if (cvStatus != kCVReturnSuccess || !outputSampleBuffer) {
        NSLog(@"Failed to create output sample buffer");
        CMVideoFormatDescriptionRelease(videoFormat);
        CVPixelBufferRelease(destYUV);
        return;
    }

    // Use for display
    // ...

    // Cleanup
    CMSampleBufferRelease(outputSampleBuffer);
    CMVideoFormatDescriptionRelease(videoFormat);
    CVPixelBufferRelease(destYUV);
}

// Don't forget to release the session when done
- (void)dealloc {
    if (_pixelTransferSession) {
        VTPixelTransferSessionRelease(_pixelTransferSession);
        _pixelTransferSession = NULL;
    }
}

Key Notes #

• YUV Format Selection: Use kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange for standard video output (matches most Apple display pipelines). If you need full dynamic range (0-255 for all channels), use kCVPixelFormatType_420YpCbCr8BiPlanarFullRange.
• Color Space Matching: Ensure the source BGRA buffer's color space matches the destination YUV buffer's. Retrieve the source color space with CVPixelBufferGetColorSpace(sourceBGRA) and add it to the destination buffer attributes if needed.
• Memory Management: Always release Core Video objects (like CVPixelBufferRef and CMSampleBufferRef)—these aren’t ARC-managed, so neglecting cleanup will cause memory leaks.

内容的提问来源于stack exchange，提问作者Bob Bobster