Swift中如何将应用音频与麦克风音频合并为单路PCM？

核心问题分析 #

当前代码仅处理了ScreenCaptureKit输出的应用音频（.audio分支），麦克风音频（.microphone分支）未实现任何逻辑，且缺少两路音频的同步合并机制。要完成需求，需解决三个核心问题：

• 两路音频的格式统一（采样率、通道数、位深必须一致）
• 音频帧的时间同步（确保两路采样帧对齐）
• 采样数据的混合计算（叠加后避免失真并转PCM16）

分步解决方案 #

1. 补全麦克风音频处理逻辑 #

先在stream(_:didOutputSampleBuffer:of:)方法中补全麦克风音频的PCM转换与缓存逻辑：

case .microphone:
    let pcmBuffer: AVAudioPCMBuffer? = createPCMBuffer(from: sampleBuffer)
    guard let pcmBuffer else { return }
    audioQueue.sync {
        micAudioBuffers.append(pcmBuffer)
        tryMergeAudioBuffers()
    }
    break

同时修改应用音频的处理分支，改为缓存后触发合并：

case .audio:
    let pcmBuffer: AVAudioPCMBuffer? = createPCMBuffer(from: sampleBuffer)
    guard let pcmBuffer else { return }
    audioQueue.sync {
        appAudioBuffers.append(pcmBuffer)
        tryMergeAudioBuffers()
    }
    break

2. 添加音频缓存与同步结构 #

在类中添加线程安全的缓存队列和缓冲区数组，用于暂存两路音频帧：

// 线程安全队列，处理音频缓存与合并
private let audioQueue = DispatchQueue(label: "com.yourapp.audio.merge")
// 应用音频缓存
private var appAudioBuffers: [AVAudioPCMBuffer] = []
// 麦克风音频缓存
private var micAudioBuffers: [AVAudioPCMBuffer] = []

3. 实现音频合并与PCM16转换 #

编写tryMergeAudioBuffers方法，取出可匹配的音频帧进行混合，同时处理格式校验与削波保护：

private func tryMergeAudioBuffers() {
    guard !appAudioBuffers.isEmpty, !micAudioBuffers.isEmpty else { return }
    
    let appBuffer = appAudioBuffers.removeFirst()
    let micBuffer = micAudioBuffers.removeFirst()
    
    // 校验格式与帧长度一致性，不一致则跳过（可添加格式转换逻辑）
    guard appBuffer.format == micBuffer.format,
          appBuffer.frameLength == micBuffer.frameLength else {
        NSLog("Audio format/frame length mismatch, skip merge")
        return
    }
    
    guard let appFloatData = appBuffer.floatChannelData,
          let micFloatData = micBuffer.floatChannelData else {
        NSLog("Failed to get float channel data")
        return
    }
    
    let channelCount = Int(appBuffer.format.channelCount)
    let frameLength = Int(appBuffer.frameLength)
    
    // 创建合并后的PCM缓冲
    guard let mergedBuffer = AVAudioPCMBuffer(pcmFormat: appBuffer.format, frameCapacity: frameLength) else {
        NSLog("Failed to create merged PCM buffer")
        return
    }
    mergedBuffer.frameLength = appBuffer.frameLength
    
    guard let mergedFloatData = mergedBuffer.floatChannelData else {
        NSLog("Failed to get merged float channel data")
        return
    }
    
    // 混合采样：两路相加后限制在±1.0范围，防止削波失真
    for channel in 0..<channelCount {
        let appChannel = appFloatData[channel]
        let micChannel = micFloatData[channel]
        let mergedChannel = mergedFloatData[channel]
        
        for i in 0..<frameLength {
            let mixedSample = appChannel[i] + micChannel[i]
            mergedChannel[i] = max(-1.0, min(1.0, mixedSample))
        }
    }
    
    // 转换为PCM16并发送至服务
    if let convertedData = convertTo16BitPCM(from: mergedBuffer) {
        deepgramService.sendAudioData(convertedData)
    }
}

4. 可选：格式统一转换（处理格式不一致场景） #

如果应用音频与麦克风音频格式不同（如采样率、通道数差异），添加格式转换逻辑：

private func convertBuffer(_ buffer: AVAudioPCMBuffer, to targetFormat: AVAudioFormat) -> AVAudioPCMBuffer? {
    guard let converter = AVAudioConverter(from: buffer.format, to: targetFormat) else {
        NSLog("Failed to create audio converter")
        return nil
    }
    
    let convertedBuffer = AVAudioPCMBuffer(pcmFormat: targetFormat, frameCapacity: buffer.frameLength)!
    convertedBuffer.frameLength = buffer.frameLength
    
    let inputBlock: AVAudioConverterInputBlock = { inNumPackets, outStatus in
        outStatus.pointee = .haveData
        return buffer
    }
    
    var error: NSError?
    converter.convert(to: convertedBuffer, error: &error, withInputFrom: inputBlock)
    
    if let error = error {
        NSLog("Conversion error: \(error.localizedDescription)")
        return nil
    }
    
    return convertedBuffer
}

关键注意事项 #

• 时间同步优化：可通过CMSampleBufferGetPresentationTimeStamp获取帧的时间戳，调整缓存策略确保两路音频的时间对齐。
• 内存管理：设置缓存的最大帧数量，超过时丢弃旧帧，避免内存溢出。
• 削波保护：混合采样时必须限制值在±1.0范围内，否则转换为PCM16会出现严重失真。

内容的提问来源于stack exchange，提问作者Garfield is on board.