AVAssetWriter录制4K视频时CIContext.render性能优化及Metal适配问题

一、先优化CIImage渲染流程（无需切换到Metal） #

如果不想立刻切换到Metal，可以先尝试以下优化，看看能否将单帧耗时压到16.67ms以内（60FPS的单帧时间上限）：

• 配置高性能CIContext：用Metal设备初始化CIContext，禁用软件渲染，指定工作色彩空间与相机输出一致，避免额外的色彩转换开销：

  let metalDevice = MTLCreateSystemDefaultDevice()
  let ciContext = CIContext(mtlDevice: metalDevice!, options: [
      .useSoftwareRenderer: false,
      .allowLowPower: false,
      .workingColorSpace: CGColorSpace(name: CGColorSpace.sRGB)!
  ])
  

• 复用CVPixelBuffer：创建CVPixelBufferPool复用缓冲区，避免每次渲染时的内存分配与释放：

  var pixelBufferPool: CVPixelBufferPool?
  let poolAttributes = [
      kCVPixelBufferPoolMinimumBufferCountKey: 3,
      kCVPixelBufferWidthKey: 3840,
      kCVPixelBufferHeightKey: 2160,
      kCVPixelBufferPixelFormatTypeKey: kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange
  ] as CFDictionary
  CVPixelBufferPoolCreate(kCFAllocatorDefault, nil, poolAttributes, &pixelBufferPool)
  

  渲染时从池中获取buffer：

  var pixelBuffer: CVPixelBuffer?
  CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, pixelBufferPool, &pixelBuffer)
  guard let buffer = pixelBuffer else { return }
  ciContext.render(compositedImage, to: buffer)
  

• 避免格式转换：直接渲染到YUV格式的CVPixelBuffer（与相机输出格式一致），不要转成RGB格式，减少数据量和转换开销。

二、Metal直接处理（最优性能方案） #

如果CIImage优化后仍不满足需求，Metal是必然选择，以下是具体实现思路：

1. 处理YUV格式的相机帧 #

相机输出的kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange包含两个平面：亮度(Y)和色度(CbCr)，无需合并成RGB，直接在YUV空间处理更高效：

• 创建Metal纹理缓存：提前初始化CVMetalTextureCache，用于快速将CVImageBuffer转为MTLTexture：

  var metalTextureCache: CVMetalTextureCache?
  CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, metalDevice!, nil, &metalTextureCache)
  

• 从CVImageBuffer生成Y/CbCr纹理：

  func createMetalTextures(from imageBuffer: CVImageBuffer) -> (YTexture: MTLTexture?, CbCrTexture: MTLTexture?) {
      let width = CVPixelBufferGetWidth(imageBuffer)
      let height = CVPixelBufferGetHeight(imageBuffer)
      
      // 创建Y平面纹理
      var yTexture: CVMetalTexture?
      CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                metalTextureCache!,
                                                imageBuffer,
                                                nil,
                                                .r8Unorm,
                                                width,
                                                height,
                                                0,
                                                &yTexture)
      let mtlYTexture = CVMetalTextureGetTexture(yTexture!)
      
      // 创建CbCr平面纹理
      var cbCrTexture: CVMetalTexture?
      CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                metalTextureCache!,
                                                imageBuffer,
                                                nil,
                                                .rg8Unorm,
                                                width/2,
                                                height/2,
                                                1,
                                                &cbCrTexture)
      let mtlCbCrTexture = CVMetalTextureGetTexture(cbCrTexture!)
      
      return (mtlYTexture, mtlCbCrTexture)
  }
  

2. Metal Shader处理帧（含水印/文字叠加） #

编写Metal着色器，直接在YUV空间处理，同时修正图像反转问题：

#include <metal_stdlib>
using namespace metal;

struct VertexIn {
    float4 position [[position]];
    float2 texCoord;
};

fragment float4 yuvToRgbAndProcess(VertexIn in [[stage_in]],
                                   texture2d<float, access::sample> yTexture,
                                   texture2d<float, access::sample> cbCrTexture,
                                   constant float4 &watermarkColor) {
    // 修正纹理坐标，避免图像反转
    float2 uv = float2(in.texCoord.x, 1.0 - in.texCoord.y);
    
    // 采样YUV像素
    float y = yTexture.sample(sampler(coord::normalized), uv).r;
    float2 cbCr = cbCrTexture.sample(sampler(coord::normalized), uv * float2(0.5, 0.5)).rg;
    
    // YUV转RGB（若不需要RGB输出，可直接在YUV空间叠加水印，进一步减少计算）
    float3 rgb = float3(
        y + 1.402 * (cbCr.g - 0.5),
        y - 0.34414 * (cbCr.r - 0.5) - 0.71414 * (cbCr.g - 0.5),
        y + 1.772 * (cbCr.r - 0.5)
    );
    
    // 叠加水印示例（右上角区域）
    if (uv.x > 0.8 && uv.y < 0.2) {
        rgb = mix(rgb, watermarkColor.rgb, watermarkColor.a);
    }
    
    return float4(rgb, 1.0);
}

3. 与AVAssetWriter配合 #

• 渲染到可写入的CVPixelBuffer：创建复用的CVPixelBufferPool，将Metal处理后的纹理渲染到buffer中：

  func renderToPixelBuffer(commandBuffer: MTLCommandBuffer, texture: MTLTexture, pixelBuffer: CVPixelBuffer) {
      CVPixelBufferLockBaseAddress(pixelBuffer, [])
      let destinationTexture = metalDevice!.makeTexture(descriptor: MTLTextureDescriptor.texture2DDescriptor(
          pixelFormat: .bgra8Unorm,
          width: CVPixelBufferGetWidth(pixelBuffer),
          height: CVPixelBufferGetHeight(pixelBuffer),
          mipmapped: false
      ))!
      destinationTexture.replace(region: MTLRegionMake2D(0, 0, destinationTexture.width, destinationTexture.height),
                                mipmapLevel: 0,
                                withBytes: CVPixelBufferGetBaseAddress(pixelBuffer)!,
                                bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer))
      
      // 执行渲染命令（需提前初始化顶点缓冲区、pipelineState等资源）
      let renderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptor)!
      renderEncoder.setRenderPipelineState(pipelineState)
      renderEncoder.setFragmentTexture(texture, index: 0)
      renderEncoder.drawPrimitives(type: .triangleStrip, vertexStart: 0, vertexCount: 4)
      renderEncoder.endEncoding()
      
      commandBuffer.commit()
      commandBuffer.waitUntilCompleted()
      CVPixelBufferUnlockBaseAddress(pixelBuffer, [])
  }
  

• 包装成CMSampleBuffer写入：将处理后的CVPixelBuffer加上原帧的时间戳，打包成CMSampleBuffer，写入AVAssetWriterInput：

  func writeSampleBuffer(pixelBuffer: CVPixelBuffer, presentationTime: CMTime) {
      var sampleBuffer: CMSampleBuffer?
      let timingInfo = CMSampleTimingInfo(duration: .invalid, presentationTimeStamp: presentationTime, decodeTimeStamp: .invalid)
      CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault,
                                        pixelBuffer,
                                        true,
                                        nil,
                                        nil,
                                        formatDescription,
                                        &timingInfo,
                                        &sampleBuffer)
      if let buffer = sampleBuffer, videoWriterInput.isReadyForMoreMediaData {
          videoWriterInput.append(buffer)
      }
  }
  

4. 额外性能优化 #

• 异步流水线处理：将相机捕获、Metal处理、AVAssetWriter写入放到不同队列，形成流水线，避免单队列阻塞。
• 减少资源创建开销：提前初始化MTLCommandQueue、MTLRenderPipelineState、CVPixelBufferPool等资源，不要在每帧处理时创建。
• 关闭Metal调试选项：发布版本中关闭Xcode的Metal API Validation，减少性能损耗。

三、解决图像反转问题 #

不要通过videoWriterInput.transform修正反转，这会增加AVAssetWriter的额外处理开销，直接在Metal着色器中修改纹理坐标（如上述Shader中的float2 uv = float2(in.texCoord.x, 1.0 - in.texCoord.y);），从根源解决反转问题，不影响性能。

内容的提问来源于stack exchange，提问作者user924