使用AVAssetWriter与CoreML处理相机流时录制帧率不一致问题咨询

你遇到的FPS下降问题核心来自两个逻辑冲突：

• AVCaptureOutput的captureOutput回调默认运行在高优先级串行队列，要求尽可能快的处理完帧返回，你当前把动作识别的同步计算逻辑直接放在回调里执行，计算耗时阻塞了后续的帧写入操作，最终导致整体帧率被拉低到30
• 动作识别本身不需要100FPS的输入速率，人体动作变化的时间粒度远大于10ms，全量帧检测属于无效算力浪费

1. 队列隔离，异步执行检测逻辑 #

单独创建动作识别专属队列，把识别逻辑从捕获队列中剥离，避免阻塞帧写入流程：

// 全局声明识别队列
private let predictionQueue = DispatchQueue(label: "com.yourapp.actionPrediction", qos: .userInitiated, attributes: .concurrent)

2. 加跳帧逻辑，降低识别帧率 #

每3~4帧才执行一次检测，既保证识别准确率，又大幅降低算力消耗：

// 全局声明帧计数变量
private var frameCount = 0
private let predictionFrameInterval = 3 // 每3帧跑一次识别，对应约33FPS识别速率

3. 优化predictor配置，启用硬件加速 #

初始化ActionClassifier时指定计算单元，优先调用Neural Engine或者GPU做推理，降低CPU占用：

let config = MLModelConfiguration()
config.computeUnits = .neuralEngine // 优先神经网络引擎，没有的话自动回退到GPU/CPU
let predictor = try ActionClassifier(configuration: config)

修改后的核心回调代码 #

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
        
    bufferImage = sampleBuffer
    guard let calibrationData = CMGetAttachment(sampleBuffer, key: kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix, attachmentModeOut: nil) as? Data else {
        return
    }
    
    cameraCalibrationMatrix = calibrationData.withUnsafeBytes { $0.pointee }
    
    // 识别逻辑丢到独立队列异步执行
    if self.isPredictorActivated == true {
        frameCount += 1
        guard frameCount % predictionFrameInterval == 0 else {
            // 跳帧，不执行识别
        } else {
            // 先retain sampleBuffer，防止异步执行时被系统回收
            guard let bufferedSample = CMSampleBufferCreateCopy(allocator: kCFAllocatorDefault, sampleBuffer: sampleBuffer) else { return }
            predictionQueue.async { [weak self] in
                defer {
                    // 用完释放sampleBuffer
                    CMSampleBufferInvalidate(bufferedSample)
                }
                do  {
                    guard let self = self else { return }
                    let poses = try self.predictor.processFrame(bufferedSample)
                    if (self.predictor.isReadyToMakePrediction) {
                        let prediction =  try self.predictor.makePrediction()
                        let confidence = prediction.confidence * 100
                        DispatchQueue.main.async {
                            self.predictionLabel.text = prediction.label + " " + String(confidence.rounded(toPlaces: 0))
                            if (prediction.label == "HandsUp" && prediction.confidence > 0.85) {
                                print("Challenging")
                                self.didChallengeVideo()
                            }
                        }
                    }
                } catch {
                    print(error.localizedDescription)
                }
            }
        }
    }
    
    // 帧写入逻辑保留在原队列，保证最高优先级执行
    let presentationTimeStamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
    
    if assetWriter == nil {
        createWriterInput(for: presentationTimeStamp)
    } else {
        let chunkDuration = CMTimeGetSeconds(CMTimeSubtract(presentationTimeStamp, chunkStartTime))
        
        if chunkDuration > 1500 || isChallenging {
            assetWriter.endSession(atSourceTime: presentationTimeStamp)
            
            let newChunkURL = chunkOutputURL!
            let chunkAssetWriter = assetWriter!
            
            chunkAssetWriter.finishWriting {
                print("finishWriting says: \(chunkAssetWriter.status.rawValue) \(String(describing: chunkAssetWriter.error))")
                print("queuing \(newChunkURL)")
                print("Chunk Duration: \(chunkDuration)")
                
                let asset = AVAsset(url: newChunkURL)
                
                print("FPS of CHUNK \(asset.tracks.first?.nominalFrameRate)")
                
                if self.isChallenging {
                    self.challengeVideoProcess(video: asset)
                }
                
                self.isChallenging = false
            }
            createWriterInput(for: presentationTimeStamp)
        }
    }
    
    if !assetWriterInput.append(sampleBuffer) {
        print("append says NO: \(assetWriter.status.rawValue) \(String(describing: assetWriter.error))")
    }
}

按照以上方案修改后，即可保证录制帧率稳定在100FPS，同时动作识别也可以正常运行。

内容的提问来源于stack exchange，提问作者Uzir Test