如何将客户端WebRTC音频流传输至Google Speech-to-Text服务

我已经实现了浏览器端通过WebRTC采集麦克风音频，将音频以包含Base64字符串的对象实时发送到Node.js的Socket.io服务器，这部分数据接收正常（日志能看到传入的音频数据）。但现在不知道怎么把收到的音频流发送给Google Cloud Speech-to-Text API完成实时转写。

我已经成功运行了Google Cloud语音文档中的快速启动应用，它能将本地麦克风的数据流传输到Speech API实现转写，但该应用依赖node-record-lpcm16和SoX工具采集音频并通过管道传给API。我知道可以通过SpeechClient的streamingRecognize方法发送音频流，请求里的audio_content应该是传入音频的位置，但不确定具体怎么对接Socket.io收到的Base64数据。

现有服务端代码 #

let io = require('socket.io')(3000, {
  cors: {origin: ['http://localhost:8080']},
})

const speech = require('@google-cloud/speech');
  
// Create a speech client
const client = new speech.SpeechClient();


const encoding = 'LINEAR16';
const sampleRateHertz = 16000;
const languageCode = 'en-US';

//speech client request header
const request = {
  config: {
    encoding: encoding,
    sampleRateHertz: sampleRateHertz,
    languageCode: languageCode,
    enableAutomaticPunctuation: true,
  },
  interimResults: false, // If you want interim results, set this to true
};

  // Create a recognize stream, this makes a request and waits for response (transcription)
    const recognizeStream = client
    .streamingRecognize(request) //send request passed to streamingRecognize method
    .on('error', console.error) //throw error if error returned
    .on('data', data =>
    {
      console.log(data.results[0].alternatives[0].words)
      process.stdout.write(
          
        data.results[0] && data.results[0].alternatives[0]
          ? `Transcription: ${data.results[0].alternatives[0].transcript}\n`
          : '\n\nReached transcription time limit, press Ctrl+C\n'
      )
    }
    );



//Create socket and listen for audio stream from webRTC

io.on('connection', socket => {
  console.log(socket.id)

  //TODO: how to send this stream to google speech?
  socket.on('audioStream', (obj) => {
      //obj is JSON object structured like this: {"audio_data": base64 string....}
     
      //verified here that stream is being received continuously
      console.log(obj)
      
  })
 
})

console.log('socket server running')

前端WebRTC发送代码 #

socket.emit('audioStream', 
  { audio_data: base64data.split('base64,')[1]}
)

解决方案 #

关键要点 #

1. 每个Socket连接对应独立的识别流：不能全局共用一个recognizeStream，否则多用户连接时会互相干扰，需要在每个socket建立连接时创建新的流，断开时销毁。
2. Base64转二进制Buffer：Google Speech-to-Text的streamingRecognize流接收的是二进制音频数据，所以要把收到的Base64字符串转换成Buffer。
3. 确保音频格式匹配：确认WebRTC采集的音频格式和Speech API配置一致（LINEAR16编码、16000Hz采样率、单声道），如果前端采集的是其他格式（比如OPUS），需要先转换格式，否则转写会失败。

修改后的服务端代码 #

let io = require('socket.io')(3000, {
  cors: {origin: ['http://localhost:8080']},
})

const speech = require('@google-cloud/speech');
const client = new speech.SpeechClient();

const encoding = 'LINEAR16';
const sampleRateHertz = 16000;
const languageCode = 'en-US';

// 连接建立时创建专属识别流
io.on('connection', socket => {
  console.log(`Client connected: ${socket.id}`);

  // 为当前socket创建识别请求和流
  const request = {
    config: {
      encoding: encoding,
      sampleRateHertz: sampleRateHertz,
      languageCode: languageCode,
      enableAutomaticPunctuation: true,
    },
    interimResults: false, // 需要实时中间结果设为true
  };

  const recognizeStream = client
    .streamingRecognize(request)
    .on('error', err => {
      console.error(`Stream error for ${socket.id}:`, err);
      socket.disconnect();
    })
    .on('data', data => {
      const result = data.results[0]?.alternatives[0];
      if (result) {
        console.log(`[${socket.id}] Transcription: ${result.transcript}`);
        // 可将转写结果回传给前端
        socket.emit('transcription', result.transcript);
      } else {
        console.log(`[${socket.id}] Transcription time limit reached`);
      }
    })
    .on('end', () => {
      console.log(`[${socket.id}] Recognize stream ended`);
    });

  // 接收音频流并转发到Google Speech
  socket.on('audioStream', (obj) => {
    try {
      // 将Base64字符串转成Buffer
      const audioBuffer = Buffer.from(obj.audio_data, 'base64');
      // 写入识别流
      recognizeStream.write(audioBuffer);
    } catch (err) {
      console.error(`Failed to process audio for ${socket.id}:`, err);
    }
  });

  // 连接断开时销毁识别流
  socket.on('disconnect', () => {
    console.log(`Client disconnected: ${socket.id}`);
    recognizeStream.end();
  });
})

console.log('Socket server running on port 3000')

额外注意事项 #

• 如果前端WebRTC采集的音频格式不是LINEAR16，需要在前端或服务端进行格式转换。比如用audioContext在浏览器中将音频转成16kHz的PCM数据，再转成Base64发送。
• 处理流的错误：当识别流出现错误时，及时断开socket并清理资源，避免内存泄漏。
• 可以将转写结果通过Socket.io回传给前端，实现实时展示。

内容的提问来源于stack exchange，提问作者Imari Childress