iOS开发：处理后CIImage转CMSampleBuffer实现视频录制存储

• 开发参考苹果官方示例《Applying Matte Effects to People in Images and Video》（官方文档地址），核心需求为将经过人像分割滤镜处理的摄像头实时视频保存为本地文件
• 已完成开发环节：摄像头原始帧采集、基于Vision的人像分割逻辑、处理后效果CIImage生成
• 卡点：官方示例未提供处理后帧落盘存储的实现，无法将生成的CIImage正确转换为可传入AVAssetWriter录制会话的CMSampleBuffer，多次尝试转换均无法正常生成可播放的处理后视频

当前实现代码如下：

extension ViewController: AVCaptureVideoDataOutputSampleBufferDelegate {

    func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
        //1. 读取摄像头输出的像素帧
        guard let pixelBuffer = sampleBuffer.imageBuffer else { return }

        //2. 生成带人像分割效果的处理后CIImage
        guard let image = processVideoFrame(pixelBuffer, sampleBuffer: sampleBuffer) else { return }

        //3. 待实现：将处理后的CIImage转换为可用的CMSampleBuffer
        let updatedBuffer = image.convertToCMSampleBuffer()

        //4. 将帧写入当前录制会话
        self.assetWriterHelper?.captureOutput(output, didOutput: sampleBuffer, from: connection)
    }
}

现有代码存在两个核心错误：

1. 最终写入录制会话时传入的仍然是原始未处理的sampleBuffer，而非转换后的处理帧，就算转换逻辑正确也会存储原始无效果视频
2. 通用的CIImage直接转CMSampleBuffer方法没有复用帧缓存、没有保留原始帧的时序信息，会导致录制花屏、音画不同步、内存暴涨、视频无法正常播放

实现步骤 #

1. 提前初始化全局复用的渲染资源 #

不要在每帧回调里创建渲染相关资源，否则会导致严重的性能问题，在视图加载、摄像头配置完成后提前初始化即可：

import Metal
import CoreVideo
import CoreMedia

class ViewController: UIViewController {
    // 基于Metal加速的CIContext，全局单例复用
    private let ciContext = CIContext(
        mtlDevice: MTLCreateSystemDefaultDevice()!,
        options: [.useSoftwareRenderer: false]
    )
    private var pixelBufferPool: CVPixelBufferPool?
    
    // 在摄像头输出分辨率确定后调用，创建像素缓冲池复用内存
    func setupPixelBufferPool(outputWidth: Int, outputHeight: Int) {
        let poolAttributes: [String: Any] = [
            kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA,
            kCVPixelBufferWidthKey as String: outputWidth,
            kCVPixelBufferHeightKey as String: outputHeight,
            kCVPixelBufferIOSurfacePropertiesKey as String: [:] // 开启硬件加速支持
        ]
        CVPixelBufferPoolCreate(nil, nil, poolAttributes as CFDictionary, &pixelBufferPool)
    }
}

2. 实现带时序信息的CIImage转CMSampleBuffer方法 #

转换时必须保留原始帧的时间戳、时长信息，同时从缓冲池复用内存避免频繁分配：

extension CIImage {
    func convertToValidCMSampleBuffer(
        originalBuffer: CMSampleBuffer,
        context: CIContext,
        bufferPool: CVPixelBufferPool
    ) -> CMSampleBuffer? {
        // 从缓冲池取出可复用的像素缓冲
        var outputPixelBuffer: CVPixelBuffer?
        let poolStatus = CVPixelBufferPoolCreatePixelBuffer(nil, bufferPool, &outputPixelBuffer)
        guard poolStatus == kCVReturnSuccess, let validBuffer = outputPixelBuffer else {
            return nil
        }
        
        // 将处理后的CIImage渲染到目标像素缓冲
        context.render(
            self,
            to: validBuffer,
            bounds: self.extent,
            colorSpace: CGColorSpaceCreateDeviceRGB()
        )
        
        // 读取原始帧的时序信息（时间戳、时长），避免音画不同步
        var sampleTiming = CMSampleTimingInfo()
        let timingStatus = CMSampleBufferGetSampleTimingInfo(
            originalBuffer,
            at: 0,
            timingInfoOut: &sampleTiming
        )
        guard timingStatus == noErr else { return nil }
        
        // 生成新的视频格式描述
        var formatDesc: CMVideoFormatDescription?
        CMVideoFormatDescriptionCreateForImageBuffer(
            allocator: nil,
            imageBuffer: validBuffer,
            formatDescriptionOut: &formatDesc
        )
        guard let validFormat = formatDesc else { return nil }
        
        // 组装最终可写入的CMSampleBuffer
        var resultBuffer: CMSampleBuffer?
        let createStatus = CMSampleBufferCreateReadyWithImageBuffer(
            allocator: nil,
            imageBuffer: validBuffer,
            formatDescription: validFormat,
            sampleTiming: &sampleTiming,
            sampleBufferOut: &resultBuffer
        )
        return createStatus == noErr ? resultBuffer : nil
    }
}

3. 修正摄像头帧回调逻辑 #

增加失败兜底逻辑，避免处理失败导致录制断帧，注意传入处理后的buffer到录制会话：

extension ViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
    func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
        // 1. 读取摄像头原始像素帧
        guard let pixelBuffer = sampleBuffer.imageBuffer else { return }
        
        // 2. 生成经过人像分割处理的CIImage
        guard let processedImage = processVideoFrame(pixelBuffer, sampleBuffer: sampleBuffer),
              let bufferPool = pixelBufferPool,
              // 3. 转换为可写入的有效CMSampleBuffer
              let updatedBuffer = processedImage.convertToValidCMSampleBuffer(
                originalBuffer: sampleBuffer,
                context: ciContext,
                bufferPool: bufferPool
              ) else {
            // 处理/转换失败时兜底写入原始帧，避免录制中断
            self.assetWriterHelper?.captureOutput(output, didOutput: sampleBuffer, from: connection)
            return
        }
        
        // 4. 写入处理后的帧到录制会话
        self.assetWriterHelper?.captureOutput(output, didOutput: updatedBuffer, from: connection)
    }
}

注意事项 #

• CIContext必须全局复用，每帧新建会导致内存暴涨、渲染帧率暴跌
• CVPixelBufferPool的宽高、像素格式必须和AVAssetWriter配置的输入格式完全一致，推荐统一用kCVPixelFormatType_32BGRA格式，兼容性最好
• 不要修改原始帧的CMSampleTimingInfo，否则会出现音画不同步、视频时长异常、无法拖动进度条的问题
• 渲染时指定的颜色空间要和摄像头输出的颜色空间匹配，避免出现颜色偏色、过饱和问题
• 帧处理逻辑不要阻塞摄像头输出的串行队列，若单帧处理耗时过长可适当降低摄像头采集帧率

内容的提问来源于stack exchange，提问作者royherma