Concentus编码解码后VOIP多路音频混音失真问题求助

问题背景 #

实现简易VOIP群聊系统，无编解码器时3-5人同时通话混音清晰正常；启用Concentus（Opus）或G722编解码器后，一对一通话无异常，但多路解码音频混音时出现大量噪音与失真。当前音频参数：48kHz采样率、40ms帧长、16位单声道。

编码端代码 #

private void AudioDataAvailable(object? sender, WaveInEventArgs e)
{
    if (IsDeafened || IsMuted)
        return;

    float max = 0;
    // interpret as 16 bit audio
    for (int index = 0; index < e.BytesRecorded; index += 2)
    {
        short sample = (short)((e.Buffer[index + 1] << 8) |
                                e.Buffer[index + 0]);
        // to floating point
        var sample32 = sample / 32768f;
        // absolute value 
        if (sample32 < 0) sample32 = -sample32;
        if (sample32 > max) max = sample32;
    }

    if (max > 0.08)
    {
        RecordDetection = DateTime.UtcNow;
    }

    if (DateTime.UtcNow.Subtract(RecordDetection).Seconds < 1)
    {
        short[] pcm = BytesToShorts(e.Buffer, 0, e.BytesRecorded);
        byte[] encoded = new byte[1000];
        Encoder.Encode(pcm, 0, 960, encoded, 0, encoded.Length);
        var voicePacket = new VoicePacket()
        {
            PacketAudio = encoded,
            PacketDataIdentifier = PacketIdentifier.Audio,
            PacketVersion = Network.Network.Version,
            PacketBytesRecorded = e.BytesRecorded
        };
        VCClient.Send(voicePacket);
    }
}

解码端代码 #

private Task VC_OnAudioReceived(byte[] Audio, string Key, float Volume, int BytesRecorded, float RotationSource)
{
    _ = Task.Factory.StartNew(() =>
    {
        var participant = Participants.FirstOrDefault(x => x.LoginKey == Key);
        if (participant != null)
        {
            try
            {
                short[] decoded = new short[960];
                Decoder.Decode(Audio, 0, Audio.Length, decoded, 0, 960, false);
                byte[] decodedBytes = ShortsToBytes(decoded, 0, decoded.Length);
                participant.FloatProvider.Volume = Volume;
                if (DirectionalAudio)
                {
                    participant.MonoToStereo.LeftVolume = (float)(0.5 + Math.Sin(RotationSource) * 0.5);
                    participant.MonoToStereo.RightVolume = (float)(0.5 - Math.Sin(RotationSource) * 0.5);
                }
                participant.WaveProvider.AddSamples(decodedBytes, 0, BytesRecorded);
            }
            catch { }
        }
    });

    if(!Stopping && AudioPlayer.PlaybackState == PlaybackState.Stopped)
        AudioPlayer.Play();

    return Task.CompletedTask;
}

关键辅助代码片段 #

ParticipantModel定义

public class ParticipantModel
{
   public string Name { get; set; } = "";
   public string LoginKey { get; set; } = "";
   public BufferedWaveProvider WaveProvider { get; set; }
   public Wave16ToFloatProvider FloatProvider { get; set; }
   public MonoToStereoSampleProvider MonoToStereo { get; set; }
}

参与者初始化

case VCSignalling_Packet.PacketIdentifier.Login:
   var participant = new ParticipantModel()
   {
      LoginKey = packet.PacketLoginKey,
      Name = packet.PacketName,
      WaveProvider = new BufferedWaveProvider(VoipService.GetRecordFormat) { DiscardOnBufferOverflow = true }
   };
   participant.FloatProvider = new Wave16ToFloatProvider(participant.WaveProvider);
   participant.MonoToStereo = new NAudio.Wave.SampleProviders.MonoToStereoSampleProvider(participant.FloatProvider.ToSampleProvider());

   OnParticipantLogin?.Invoke(participant);
   break;

音频格式与编解码器初始化

// 音频格式定义
public static WaveFormat GetRecordFormat { get => new WaveFormat(SampleRate, 16, Channels); }
public static WaveFormat GetAudioFormat { get => WaveFormat.CreateIeeeFloatWaveFormat(SampleRate, Channels * 2); }

// Opus编解码器初始化
Encoder = new OpusEncoder(SampleRate, Channels, Concentus.Enums.OpusApplication.OPUS_APPLICATION_VOIP);
Encoder.Complexity = 0;
Encoder.UseVBR = true;

Decoder = new OpusDecoder(SampleRate, Channels);

// 混音与播放初始化
Mixer = new MixingSampleProvider(GetAudioFormat);
Normalizer = new SoftLimiter(Mixer);
Normalizer.Boost.CurrentValue = 5;

AudioRecorder = audioManager.CreateRecorder(GetRecordFormat);
AudioPlayer = audioManager.CreatePlayer(Normalizer);

核心成因 #

1. 编解码帧长不匹配
   48kHz采样率下，40ms帧对应的采样点数应为48000 * 0.04 = 1920，但当前编码和解码都硬编码为960采样点，仅处理了一半的音频数据，导致帧数据截断/重复，混音时出现时序混乱与失真。

2. 解码后数据长度传递错误
   解码生成的是960个short（1920字节），但调用AddSamples时传入了原始录制的BytesRecorded（3840字节），这会导致读取超出解码数组范围的垃圾数据，引入大量噪音。

3. Opus编码器配置不合理
   Complexity设为0会严重降低编码质量，低复杂度下Opus对语音帧的边缘处理存在缺陷，多路混音时这些缺陷被放大；同时VBR（可变比特率）会导致帧间质量波动，进一步加剧混音失真。

4. 混音格式不匹配
   GetAudioFormat定义为Channels * 2通道，若原始Channels为1，实际生成的是4通道格式，但参与者的MonoToStereo输出是2通道，格式不匹配会导致混音时数据解析错误。

修复方案 #

1. 修正编解码帧长 #

将编码和解码的采样点数改为1920，匹配40ms帧长：

• 编码端：

  Encoder.Encode(pcm, 0, 1920, encoded, 0, encoded.Length);
  

• 解码端：

  short[] decoded = new short[1920];
  Decoder.Decode(Audio, 0, Audio.Length, decoded, 0, 1920, false);
  

2. 修复解码数据长度传递 #

使用解码后的实际字节数调用AddSamples，避免读取垃圾数据：

byte[] decodedBytes = ShortsToBytes(decoded, 0, decoded.Length);
int actualByteCount = decodedBytes.Length; // 1920*2=3840，与原始录制长度一致
participant.WaveProvider.AddSamples(decodedBytes, 0, actualByteCount);

3. 优化Opus编码器配置 #

调整参数平衡质量与性能：

Encoder.Complexity = 6; // 取值5-8，推荐6
Encoder.UseVBR = false;
Encoder.UseCVBR = true; // 恒定比特率减少帧间波动

4. 统一混音格式 #

修正GetAudioFormat为标准立体声浮点格式：

public static WaveFormat GetAudioFormat { get => WaveFormat.CreateIeeeFloatWaveFormat(SampleRate, 2); }

关于降噪处理 #

当前噪音与失真核心是编解码帧不匹配和数据越界导致的结构性失真，优先修复上述问题即可解决。若修复后仍存在环境背景噪音，可添加轻量降噪模块，但并非当前问题的必要解决方案。

内容的提问来源于stack exchange，提问作者SineVector241