Unity中如何替换NAudio，用内置AudioClip播放IBM Watson TTS流式音频？

我目前通过WebSocket从IBM Watson TTS流式传输音频，已经用NAudio实现了流式播放，但为了兼容Oculus Lipsync这类依赖AudioClip的插件，需要将NAudio替换为Unity内置的AudioClip方案。

当前NAudio实现方案 #

1. WebSocket接收字节写入MemoryStream #

private MemoryStream ms = new MemoryStream();

websocket.OnMessage += async (bytes) =>
{
    var pos = ms.Position;
    ms.Position = ms.Length;
    ms.Write(bytes, 0, bytes.Length);
    ms.Position = pos;

    if (!isPlaying)
    {
        StartCoroutine(PlayAudioStream());
    }
};

2. NAudio处理音频播放 #

using (var blockAlignedStream = new BlockAlignReductionStream(WaveFormatConversionStream.CreatePcmStream(new RawSourceWaveStream(ms, new WaveFormat(22050, 16, 1)))))
{
    var aggregator = new SampleAggregator(blockAlignedStream.ToSampleProvider());
    aggregator.NotificationCount = blockAlignedStream.WaveFormat.SampleRate / 50;
    using (var wo = new WaveOutEvent())
    {
        isPlaying = true;
        wo.Init(aggregator);
        wo.Play();

        while (wo.PlaybackState == PlaybackState.Playing) //&& !disconnected
        {
            yield return new WaitForEndOfFrame();
        }
        wo.Dispose();
        aggregator.Reset();
        aggregator = null;
    }
}

替换为Unity AudioClip的实现方案 #

针对Watson TTS返回的22050采样率、16位单声道PCM格式，我们可以通过Unity动态AudioClip配合AudioSource实现流式播放，同时兼容依赖AudioClip的插件：

1. 定义核心变量与初始化 #

private MemoryStream audioStream = new MemoryStream();
private object streamLock = new object(); // 线程安全锁，避免WebSocket回调与音频读取冲突
private AudioSource audioSource;
private const int sampleRate = 22050; // 与Watson TTS输出格式匹配
private const int channelCount = 1;
private const int bytesPerSample = 2; // 16位PCM每个样本占2字节

void Awake()
{
    // 获取或创建AudioSource组件
    audioSource = GetComponent<AudioSource>() ?? gameObject.AddComponent<AudioSource>();
    // 创建动态音频剪辑，播放时会主动调用OnAudioRead获取数据
    audioSource.clip = AudioClip.Create("WatsonTTSStream", sampleRate * 5, channelCount, sampleRate, true, OnAudioRead);
}

2. 调整WebSocket接收逻辑 #

websocket.OnMessage += (bytes) =>
{
    lock(streamLock)
    {
        // 将新接收的音频字节追加到流末尾
        var currentPos = audioStream.Position;
        audioStream.Position = audioStream.Length;
        audioStream.Write(bytes, 0, bytes.Length);
        audioStream.Position = currentPos;
    }

    // 若未播放则启动播放协程
    if (!isPlaying)
    {
        StartCoroutine(PlayStreamCoroutine());
    }
};

3. 音频数据读取回调 #

Unity播放动态AudioClip时，会持续调用此方法获取音频样本数据，我们需要把16位PCM字节转换成Unity支持的float数组（范围-1到1）：

private void OnAudioRead(float[] data)
{
    lock(streamLock)
    {
        int requiredBytes = data.Length * bytesPerSample;
        byte[] byteBuffer = new byte[requiredBytes];
        int actualBytesRead = audioStream.Read(byteBuffer, 0, requiredBytes);

        // 转换16位PCM到float样本
        for (int i = 0; i < actualBytesRead / bytesPerSample; i++)
        {
            short pcmSample = BitConverter.ToInt16(byteBuffer, i * bytesPerSample);
            data[i] = pcmSample / 32768f; // 16位有符号范围是-32768~32767，转成-1~1的float
        }

        // 若流中数据不足，剩余位置填充0静音
        for (int i = actualBytesRead / bytesPerSample; i < data.Length; i++)
        {
            data[i] = 0f;
        }
    }
}

4. 播放控制协程 #

private bool isPlaying = false;

private IEnumerator PlayStreamCoroutine()
{
    isPlaying = true;
    audioSource.Play();

    // 持续等待，直到播放结束且流中无剩余数据
    while (audioSource.isPlaying || (audioStream.Length - audioStream.Position) > 0)
    {
        yield return null;
    }

    audioSource.Stop();
    isPlaying = false;

    // 重置流，准备下一次TTS请求
    lock(streamLock)
    {
        audioStream.SetLength(0);
        audioStream.Position = 0;
    }
}

关键说明 #

• 动态AudioClip：AudioClip.Create创建的动态剪辑会按需从回调中获取数据，完美适配流式场景
• 线程安全：WebSocket回调运行在后台线程，音频读取在主线程，必须用锁保护MemoryStream的操作
• 格式转换：严格按照16位PCM到float的转换规则，保证音频播放质量
• Oculus Lipsync兼容：直接将此AudioSource作为Lipsync的输入源即可，因为使用了标准Unity AudioClip

内容的提问来源于stack exchange，提问作者bomanden