Flutter(Objective-C)相机：处理后返回的live-preview CVPixelBuffer无法显示

核心诱因 #

• CVPixelBuffer格式不匹配：官方Camera Plugin默认预览帧格式为kCVPixelFormatType_32BGRA，如果从UIImage转换得到的CVPixelBuffer为RGBA等其他格式，Flutter侧纹理渲染器无法识别，会直接跳过渲染，表现为预览无变化。
• 尺寸不匹配：处理后的UIImage尺寸与原相机输出的预览帧宽高不一致，Flutter纹理初始化时已经固定了渲染尺寸，不符合尺寸的帧会被直接丢弃。
• PixelBuffer属性缺失：Flutter iOS端纹理渲染要求CVPixelBuffer附带IOSurface属性才能完成高效渲染，若转换生成的PixelBuffer仅在普通内存中存储、没有配置IOSurface属性，无法被渲染到纹理上。
• UIImage方向未校正：相机输出的SampleBuffer自带旋转方向，转换为UIImage时如果没有校正方向属性，最终生成的PixelBuffer内容是旋转状态，可能渲染到可视区域外，看起来像预览未生效。

修复步骤 #

1. 先校正UIImage方向，在转换为CVPixelBuffer前，将UIImage的方向修正为向上，避免内容偏移：

- (UIImage *)fixOrientation:(UIImage *)image {
    if (image.imageOrientation == UIImageOrientationUp) return image;
    CGAffineTransform transform = CGAffineTransformIdentity;
    switch (image.imageOrientation) {
        case UIImageOrientationDown:
        case UIImageOrientationDownMirrored:
            transform = CGAffineTransformTranslate(transform, image.size.width, image.size.height);
            transform = CGAffineTransformRotate(transform, M_PI);
            break;
        case UIImageOrientationLeft:
        case UIImageOrientationLeftMirrored:
            transform = CGAffineTransformTranslate(transform, image.size.width, 0);
            transform = CGAffineTransformRotate(transform, M_PI_2);
            break;
        case UIImageOrientationRight:
        case UIImageOrientationRightMirrored:
            transform = CGAffineTransformTranslate(transform, 0, image.size.height);
            transform = CGAffineTransformRotate(transform, -M_PI_2);
            break;
        default:
            break;
    }
    switch (image.imageOrientation) {
        case UIImageOrientationUpMirrored:
        case UIImageOrientationDownMirrored:
            transform = CGAffineTransformTranslate(transform, image.size.width, 0);
            transform = CGAffineTransformScale(transform, -1, 1);
            break;
        case UIImageOrientationLeftMirrored:
        case UIImageOrientationRightMirrored:
            transform = CGAffineTransformTranslate(transform, image.size.height, 0);
            transform = CGAffineTransformScale(transform, -1, 1);
            break;
        default:
            break;
    }
    CGContextRef ctx = CGBitmapContextCreate(NULL, image.size.width, image.size.height,
                                             CGImageGetBitsPerComponent(image.CGImage), 0,
                                             CGImageGetColorSpace(image.CGImage),
                                             CGImageGetBitmapInfo(image.CGImage));
    CGContextConcatCTM(ctx, transform);
    switch (image.imageOrientation) {
        case UIImageOrientationLeft:
        case UIImageOrientationLeftMirrored:
        case UIImageOrientationRight:
        case UIImageOrientationRightMirrored:
            CGContextDrawImage(ctx, CGRectMake(0,0,image.size.height,image.size.width), image.CGImage);
            break;
        default:
            CGContextDrawImage(ctx, CGRectMake(0,0,image.size.width,image.size.height), image.CGImage);
            break;
    }
    CGImageRef cgimg = CGBitmapContextCreateImage(ctx);
    UIImage *img = [UIImage imageWithCGImage:cgimg];
    CGContextRelease(ctx);
    CGImageRelease(cgimg);
    return img;
}

2. 修正CVPixelBuffer转换逻辑，强制指定格式为BGRA，开启IOSurface配置，确保和官方原输出帧属性一致：

- (CVPixelBufferRef)pixelBufferFromCGImage:(CGImageRef)image size:(CGSize)size {
    NSDictionary *options = @{
        (__bridge NSString *)kCVPixelBufferCGImageCompatibilityKey: @(YES),
        (__bridge NSString *)kCVPixelBufferCGBitmapContextCompatibilityKey: @(YES),
        (__bridge NSString *)kCVPixelBufferIOSurfacePropertiesKey: @{},
        (__bridge NSString *)kCVPixelBufferWidthKey: @(size.width),
        (__bridge NSString *)kCVPixelBufferHeightKey: @(size.height)
    };
    CVPixelBufferRef pxbuffer = NULL;
    CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, size.width, size.height,
                                          kCVPixelFormatType_32BGRA, (__bridge CFDictionaryRef)options,
                                          &pxbuffer);
    NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL);
    CVPixelBufferLockBaseAddress(pxbuffer, 0);
    void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
    NSParameterAssert(pxdata != NULL);
    CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
    CGContextRef context = CGBitmapContextCreate(pxdata, size.width, size.height,
                                                 8, CVPixelBufferGetBytesPerRow(pxbuffer),
                                                 rgbColorSpace,
                                                 kCGImageAlphaPremultipliedFirst | kCGBitmapByteOrder32Little);
    NSParameterAssert(context);
    CGContextDrawImage(context, CGRectMake(0, 0, size.width, size.height), image);
    CGColorSpaceRelease(rgbColorSpace);
    CGContextRelease(context);
    CVPixelBufferUnlockBaseAddress(pxbuffer, 0);
    return pxbuffer;
}

3. 调整原有业务代码，使用上述两个方法替换原有转换逻辑，确保处理后的图像尺寸和原预览帧尺寸完全一致：

UIImage* image = [self imageFromSampleBuffer: sampleBuffer];
// 先校正方向
UIImage *fixedImage = [self fixOrientation:image];
InteractingResultContainer* resultContainer = [_cameraPreviewInteractor operateOnImageFrame: fixedImage];
if(resultContainer.image != nil) {
    // 强制和原图像尺寸保持一致，避免尺寸不匹配
    CVPixelBufferRef editedBuffer = [self pixelBufferFromCGImage: resultContainer.image.CGImage size:image.size];
    CFRetain(editedBuffer);
    CVPixelBufferRef old = _latestPixelBuffer;
    while (!OSAtomicCompareAndSwapPtrBarrier(old, editedBuffer, (void **)&_latestPixelBuffer)) {
        old = _latestPixelBuffer;
    }
    if (old != nil) {
        CFRelease(old);
    }
}
else {
    CVPixelBufferRef newBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    CFRetain(newBuffer);
    CVPixelBufferRef old = _latestPixelBuffer;
    while (!OSAtomicCompareAndSwapPtrBarrier(old, newBuffer, (void **)&_latestPixelBuffer)) {
        old = _latestPixelBuffer;
    }
    if (old != nil) {
        CFRelease(old);
    }
}

if (_onFrameAvailable) {
    _onFrameAvailable();
}

内容的提问来源于stack exchange，提问作者Mykhailo Bazyshyn