如何将WebSocket传来的ArrayBuffer转为MediaStream并通过addTrack传输到WebRTC服务器

核心优化思路 #

• 避免频繁创建AudioBufferSourceNode：每次新建节点会带来额外CPU开销，且易导致多音频叠加播放
• 采用流式音频处理节点：使用AudioWorklet（现代浏览器推荐，低CPU占用）或ScriptProcessorNode（兼容旧浏览器）实现持续音频注入，减少内存与CPU消耗
• 实现有序缓冲队列：确保音频数据按顺序填充播放，彻底解决多音频叠加问题

方案1：使用AudioWorklet（低CPU占用，推荐） #

AudioWorklet在独立线程处理音频，不会阻塞主线程，能显著降低CPU负载。

步骤1：创建AudioWorklet处理器脚本（如audio-inject-processor.js）

class AudioInjectProcessor extends AudioWorkletProcessor {
  constructor() {
    super();
    this.audioQueue = [];
    // 接收主线程传来的音频数据
    this.port.onmessage = (e) => this.audioQueue.push(e.data);
  }

  process(inputs, outputs) {
    const channel = outputs[0][0];
    let offset = 0;

    // 从队列取数据填充输出缓冲区
    while (offset < channel.length && this.audioQueue.length > 0) {
      const audioData = this.audioQueue[0];
      const copyLength = Math.min(channel.length - offset, audioData.length);
      
      channel.set(audioData.subarray(0, copyLength), offset);
      offset += copyLength;

      // 处理剩余数据或移除已用完的队列项
      if (audioData.length === copyLength) {
        this.audioQueue.shift();
      } else {
        this.audioQueue[0] = audioData.subarray(copyLength);
      }
    }

    // 剩余位置填充静音数据
    if (offset < channel.length) {
      channel.fill(0, offset);
    }

    return true;
  }
}

registerProcessor('audio-inject-processor', AudioInjectProcessor);

步骤2：主线程初始化与数据处理逻辑

class WsAudioToMediaStream {
  constructor(sampleRate = 48000) {
    this.sampleRate = sampleRate;
    this.audioContext = new AudioContext({ sampleRate });
    this.mediaStreamDest = this.audioContext.createMediaStreamDestination();
    this.workletNode = null;
    this.initWorklet();
  }

  async initWorklet() {
    // 加载处理器脚本
    await this.audioContext.audioWorklet.addModule('audio-inject-processor.js');
    // 创建Worklet节点，指定输出声道数
    this.workletNode = new AudioWorkletNode(
      this.audioContext,
      'audio-inject-processor',
      { outputChannelCount: [1] }
    );
    this.workletNode.connect(this.mediaStreamDest);
  }

  // 处理WebSocket接收的音频数据
  processWsData(wsAudioData) {
    if (this.audioContext.state === 'suspended') {
      this.audioContext.resume();
    }
    // 转换Uint8Array为Web Audio所需的Float32Array（根据实际音频格式调整）
    const float32Data = this.convertUint8ToFloat32(wsAudioData);
    this.workletNode.port.postMessage(float32Data);
  }

  // 获取用于WebRTC的MediaStream
  getStream() {
    return this.mediaStreamDest.stream;
  }

  // 16位PCM格式转换示例：Uint8Array（小端）转Float32Array（[-1,1]范围）
  convertUint8ToFloat32(uint8Data) {
    const int16Data = new Int16Array(uint8Data.buffer);
    const float32Data = new Float32Array(int16Data.length);
    for (let i = 0; i < int16Data.length; i++) {
      float32Data[i] = int16Data[i] / 32768;
    }
    return float32Data;
  }
}

步骤3：关联到RTCPeerConnection

// 初始化转换器实例
const audioConverter = new WsAudioToMediaStream(你的采样率);

// WebSocket接收数据时触发处理
ws.onmessage = (event) => {
  const wsAudioData = new Uint8Array(event.data);
  audioConverter.processWsData(wsAudioData);
};

// 将音频轨道添加到PeerConnection
peerConnection.addTrack(
  audioConverter.getStream().getAudioTracks()[0],
  audioConverter.getStream()
);

方案2：使用ScriptProcessorNode（兼容旧浏览器） #

若需兼容不支持AudioWorklet的浏览器，可使用ScriptProcessorNode（虽已标记为deprecated，但仍可稳定运行）：

class WsAudioToMediaStream {
  constructor(sampleRate = 48000, bufferSize = 1024) {
    this.sampleRate = sampleRate;
    this.audioContext = new AudioContext({ sampleRate });
    this.mediaStreamDest = this.audioContext.createMediaStreamDestination();
    this.audioQueue = [];
    this.initScriptProcessor(bufferSize);
  }

  initScriptProcessor(bufferSize) {
    const scriptNode = this.audioContext.createScriptProcessor(bufferSize, 1, 1);
    scriptNode.onaudioprocess = (event) => {
      const channelData = event.outputBuffer.getChannelData(0);
      let offset = 0;

      // 填充队列中的音频数据
      while (offset < channelData.length && this.audioQueue.length > 0) {
        const audioData = this.audioQueue[0];
        const copyLength = Math.min(channelData.length - offset, audioData.length);
        
        channelData.set(audioData.subarray(0, copyLength), offset);
        offset += copyLength;

        if (audioData.length === copyLength) {
          this.audioQueue.shift();
        } else {
          this.audioQueue[0] = audioData.subarray(copyLength);
        }
      }

      // 剩余位置填充静音
      if (offset < channelData.length) {
        channelData.fill(0, offset);
      }
    };
    scriptNode.connect(this.mediaStreamDest);
  }

  processWsData(wsAudioData) {
    if (this.audioContext.state === 'suspended') {
      this.audioContext.resume();
    }
    const float32Data = this.convertUint8ToFloat32(wsAudioData);
    this.audioQueue.push(float32Data);
  }

  getStream() {
    return this.mediaStreamDest.stream;
  }

  convertUint8ToFloat32(uint8Data) {
    const int16Data = new Int16Array(uint8Data.buffer);
    const float32Data = new Float32Array(int16Data.length);
    for (let i = 0; i < int16Data.length; i++) {
      float32Data[i] = int16Data[i] / 32768;
    }
    return float32Data;
  }
}

方案优势说明 #

1. 降低CPU占用：
   • AudioWorklet在独立线程处理音频，彻底避免主线程阻塞，大幅减少设备发热
   • 摒弃频繁创建AudioBufferSourceNode的低效模式，用单节点持续处理流式数据
2. 解决多音频叠加：
   • 通过缓冲队列按顺序填充音频数据，确保播放连续无重叠

额外优化建议 #

• 调整缓冲区大小：根据延迟需求选择1024/2048等数值，更大的缓冲区CPU占用更低但延迟更高
• 压缩音频解码：若WebSocket传输的是Opus等压缩格式，可在Web Worker中完成解码，避免阻塞主线程
• 队列长度控制：限制缓冲队列最大长度，避免内存溢出，队列过长时可丢弃旧数据

内容的提问来源于stack exchange，提问作者Lucas Wang