使用AVSampleBufferAudioRenderer播放Opus解码PCM流音频遇问题

编辑： 根据建议更新了代码，修复了ASBD并尝试修正PTS。目前仍无音频输出，但至少已无报错。

我正在开发一个iOS项目，接收Opus音频数据包，尝试使用AVSampleBufferAudioRenderer进行播放。当前使用Opus自带的解码器，最终只需让解码后的PCM数据包播放即可。整个流程缺乏详尽文档，但我认为已接近目标。以下是目前的代码（经过精简，部分值硬编码以简化逻辑）：

static AVSampleBufferAudioRenderer* audioRenderer;
static AVSampleBufferRenderSynchronizer* renderSynchronizer;

int samplesPerFrame = 240;
int channelCount    = 2;
int sampleRate      = 48000;
int streams         = 1;
int coupledStreams  = 1;
char mapping[8] = ['\0','\x01','\0','\0','\0','\0','\0','\0'];

CMTime startPTS;

// called when the stream is about to start
void AudioInit()
{
    renderSynchronizer = [[AVSampleBufferRenderSynchronizer alloc] init];
    audioRenderer = [[AVSampleBufferAudioRenderer alloc] init];
    [renderSynchronizer addRenderer:audioRenderer];
    
    int decodedPacketSize = samplesPerFrame * sizeof(short) * channelCount; // 240 samples per frame * 2 channels
    decodedPacketBuffer = SDL_malloc(decodedPacketSize);
    
    int err;
    opusDecoder = opus_multistream_decoder_create(sampleRate,       // 48000
                                                  channelCount,     // 2
                                                  streams,          // 1
                                                  coupledStreams,   // 1
                                                  mapping,
                                                  &err);

    [renderSynchronizer setRate:1.0 time:kCMTimeZero atHostTime:CMClockGetTime(CMClockGetHostTimeClock())];
    startPTS = CMClockGetTime(CMClockGetHostTimeClock());
}

// called every X milliseconds with a new packet of audio data to play, IF there's audio. (while testing, X = 5)
void AudioDecodeAndPlaySample(char* sampleData, int sampleLength)
{
    // decode the packet from Opus to (I think??) Linear PCM
    int numSamples;
    numSamples = opus_multistream_decode(opusDecoder,
                                         (unsigned char *)sampleData,
                                         sampleLength,
                                         (short*)decodedPacketBuffer,
                                         samplesPerFrame, // 240
                                         0);

    int bufferSize = sizeof(short) * numSamples * channelCount; // 240 samples * 2 channels

    CMTime currentPTS = CMTimeSubtract(CMClockGetTime(CMClockGetHostTimeClock()), startPTS);

    // LPCM stream description
    AudioStreamBasicDescription asbd = {
        .mFormatID          = kAudioFormatLinearPCM,
        .mFormatFlags       = kLinearPCMFormatFlagIsSignedInteger,
        .mBytesPerPacket    = sizeof(short) * channelCount,
        .mFramesPerPacket   = 1,
        .mBytesPerFrame     = sizeof(short) * channelCount,
        .mChannelsPerFrame  = channelCount, // 2
        .mBitsPerChannel    = 16,
        .mSampleRate        = sampleRate // 48000,
        .mReserved          = 0
    };
    
    // audio format description wrapper around asbd
    CMAudioFormatDescriptionRef audioFormatDesc;
    OSStatus status = CMAudioFormatDescriptionCreate(kCFAllocatorDefault,
                                                     &asbd,
                                                     0,
                                                     NULL,
                                                     0,
                                                     NULL,
                                                     NULL,
                                                     &audioFormatDesc);
    
    // data block to store decoded packet into
    CMBlockBufferRef blockBuffer;
    status = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault,
                                                decodedPacketBuffer,
                                                bufferSize,
                                                kCFAllocatorNull,
                                                NULL,
                                                0,
                                                bufferSize,
                                                0,
                                                &blockBuffer);
    
    // data block converted into a sample buffer
    CMSampleBufferRef sampleBuffer;
    status = CMAudioSampleBufferCreateReadyWithPacketDescriptions(kCFAllocatorDefault,
                                                                  blockBuffer,
                                                                  audioFormatDesc,
                                                                  numSamples,
                                                                  currentPTS,
                                                                  NULL,
                                                                  &sampleBuffer);
    
    
    // queueing sample buffer onto audio renderer
    [audioRenderer enqueueSampleBuffer:sampleBuffer];
}

AudioDecodeAndPlaySample函数来自我使用的库，如注释所述，每隔X毫秒（测试时X=5）会被调用一次，传入一份音频数据包（无音频时不会调用）。

我可能存在多处错误——我认为Opus解码器解码后输出的是交错式Linear PCM，也希望自己构建的AudioStreamBasicDescription是正确的。我不确定在CMAudioSampleBufferCreateReadyWithPacketDescriptions中该如何设置PTS（显示时间戳），尝试用当前主机时间 - 初始化主机时间来计算，但不确定是否可行。

我看到的大多数enqueueSampleBuffer示例代码都用requestMediaDataWhenReady搭配调度队列进行包裹，我也尝试过但无效果（我认为这只是最佳实践而非必需，所以先尝试最简实现；若确实必需可重新添加）。

若您更习惯Swift语言也可使用Swift回复，两种语言我都能处理（无奈我现在必须使用Objective-C 😀）。现寻求帮助排查问题，确认ASBD配置、PTS设置是否正确，以及是否遗漏关键步骤导致无音频输出。

问题排查关键点 #

1. AudioStreamBasicDescription 配置修正 #

Opus解码出的PCM是小端字节序的交错式PCM，你的ASBD缺少关键格式标记，需补充kLinearPCMFormatFlagIsPacked | kLinearPCMFormatFlagIsLittleEndian，同时建议先清零结构体避免内存脏值：

AudioStreamBasicDescription asbd;
memset(&asbd, 0, sizeof(asbd));
asbd.mFormatID          = kAudioFormatLinearPCM;
asbd.mFormatFlags       = kLinearPCMFormatFlagIsSignedInteger | kLinearPCMFormatFlagIsPacked | kLinearPCMFormatFlagIsLittleEndian;
asbd.mBytesPerPacket    = sizeof(short) * channelCount;
asbd.mFramesPerPacket   = 1;
asbd.mBytesPerFrame     = sizeof(short) * channelCount;
asbd.mChannelsPerFrame  = channelCount;
asbd.mBitsPerChannel    = 16;
asbd.mSampleRate        = sampleRate;

2. PTS 时间戳逻辑调整 #

用主机时间计算PTS会导致同步混乱，AVSampleBufferRenderSynchronizer需要的是基于采样数累加的媒体时间戳：

• 新增全局变量：CMTime currentMediaTime = kCMTimeZero;
• 每次解码后计算帧持续时间并更新时间戳：

CMTime frameDuration = CMTimeMake(numSamples, sampleRate);
CMTime currentPTS = currentMediaTime;
currentMediaTime = CMTimeAdd(currentMediaTime, frameDuration);

3. 渲染器启动与状态检查 #

• 必须调用启动方法：[renderSynchronizer startRenderingAtTime:kCMTimeZero];
• 添加状态监听排查错误：

// 在AudioInit中添加监听
[audioRenderer addObserver:self forKeyPath:@"status" options:NSKeyValueObservingOptionNew context:nil];

// 实现监听回调
- (void)observeValueForKeyPath:(NSString *)keyPath ofObject:(id)object change:(NSDictionary<NSKeyValueChangeKey,id> *)change context:(void *)context {
    if ([keyPath isEqualToString:@"status"]) {
        AVSampleBufferRendererStatus status = [change[NSKeyValueChangeNewKey] integerValue];
        if (status == AVSampleBufferRendererStatusFailed) {
            NSLog(@"Audio renderer error: %@", audioRenderer.error);
        }
    }
}

4. Core Foundation 对象内存管理 #

创建的CMAudioFormatDescriptionRef、CMBlockBufferRef、CMSampleBufferRef需手动释放，避免内存泄漏和数据异常：

// 在enqueue后添加释放逻辑
CFRelease(sampleBuffer);
CFRelease(blockBuffer);
CFRelease(audioFormatDesc);

5. requestMediaDataWhenReady 的必要性 #

AVSampleBufferAudioRenderer是按需拉取数据的模型，直接调用enqueueSampleBuffer可能导致数据堆积或丢失，正确写法：

// 在AudioInit中初始化调度队列
dispatch_queue_t audioQueue = dispatch_queue_create("com.your.app.audio.render", DISPATCH_QUEUE_SERIAL);
[audioRenderer requestMediaDataWhenReadyOnQueue:audioQueue usingBlock:^{
    while ([audioRenderer isReadyForMoreMediaData]) {
        // 此处处理解码并调用enqueueSampleBuffer，需确保数据供给速率匹配播放速率
    }
}];

内容的提问来源于stack exchange，提问作者Jason Ericson