ReactJS+TypeScript实现麦克风与扬声器同步录音并通过Socket.io传输转写

一、同步捕获麦克风与扬声器音频 #

要同时录制麦克风和系统扬声器音频，需要分别获取麦克风媒体流和系统音频流，再将两者的音频轨道合并为一个流后进行录制：

1. 获取系统音频流：使用navigator.mediaDevices.getDisplayMedia，通过指定audio: true捕获系统输出音频（部分浏览器需同时请求视频，后续可移除视频轨道）；
2. 合并音频轨道：将麦克风流和系统音频流的音频轨道添加到同一个MediaStream实例中；
3. 统一录制：用合并后的流初始化MediaRecorder，实现同步录制。

修改后的核心逻辑如下：

// 补充状态变量存储系统音频流
const [systemAudioStream, setSystemAudioStream] = useState<MediaStream | null>(null);

const startRecording = (id: string) => {
  socket.connect();
  console.log("=recording started");

  // 同时请求麦克风流和系统音频流
  Promise.all([
    navigator.mediaDevices.getUserMedia({ 
      audio: { echoCancellation: false, noiseSuppression: false } // 关闭降噪避免音频失真
    }),
    navigator.mediaDevices.getDisplayMedia({ audio: true, video: false })
  ]).then(([micStream, systemStream]) => {
    // 移除浏览器强制返回的视频轨道（若存在）
    systemStream.getVideoTracks().forEach(track => track.stop());

    // 创建合并后的媒体流
    const combinedStream = new MediaStream();
    micStream.getAudioTracks().forEach(track => combinedStream.addTrack(track));
    systemStream.getAudioTracks().forEach(track => combinedStream.addTrack(track));

    const mimeTypes = ["audio/mp4", "audio/webm"].filter(type => MediaRecorder.isTypeSupported(type));
    if (mimeTypes.length === 0) return alert("Browser not supported");

    setIsRecording(true);
    setStream(combinedStream);
    setSystemAudioStream(systemStream);
    setTimerInterval(setInterval(() => setTranscriptLength(t => t + 1), 1000));

    const recorder = new MediaRecorder(combinedStream, { mimeType: mimeTypes[0] });
    recorder.addEventListener("dataavailable", async (event) => {
      if (event.data.size > 0 && socket.connected) {
        console.log("sending audio");
        // 将Blob转为ArrayBuffer提升传输稳定性
        const arrayBuffer = await event.data.arrayBuffer();
        socket.emit("audio", { roomId: props.roomId, data: arrayBuffer });
      } else {
        console.log("no data available or socket disconnected");
      }
    });
    recorder.start(1000); // 每1秒分片发送音频数据
  }).catch(err => {
    console.error("获取媒体流失败:", err);
    alert("无法获取麦克风或系统音频权限，请检查浏览器设置");
  });
};

// 停止录制时需同时清理两个流的轨道
const stopRecording = () => {
  stream?.getTracks().forEach(track => track.stop());
  systemAudioStream?.getTracks().forEach(track => track.stop());
  setIsRecording(false);
  clearInterval(timerInterval!);
  socket.emit("stop-transcript", { roomId: props.roomId });
  console.log("recording stopped");
  setStream(null);
  setSystemAudioStream(null);
  setTranscriptLength(0);
};

注意事项： #

• 权限与环境：getDisplayMedia需要用户手动授权，且仅支持HTTPS环境或localhost；
• 兼容性：部分浏览器（如Firefox）需同时请求视频才能捕获系统音频，代码已处理移除视频轨道的逻辑；
• 音频质量：关闭麦克风的回声消除和降噪，避免合并后的音频出现失真。

二、通过Socket.io发送音频至服务器转写 #

你的现有代码已实现基础发送逻辑，以下优化可提升稳定性和效率：

1. Socket状态监听：添加连接状态监听，避免断开时发送无效数据；
2. 错误处理：捕获Socket异常，防止前端崩溃；
3. 二进制传输优化：将Blob转为ArrayBuffer，适配Socket.io的二进制传输机制。

优化后的Socket监听逻辑（组件初始化时添加）： #

useEffect(() => {
  socket.on("connect", () => console.log("Socket connected"));
  socket.on("disconnect", () => {
    console.log("Socket disconnected");
    if (isRecording) {
      alert("与服务器连接断开，已停止录制");
      stopRecording();
    }
  });
  socket.on("error", (err) => console.error("Socket error:", err));

  // 组件卸载时清理监听
  return () => {
    socket.off("connect");
    socket.off("disconnect");
    socket.off("error");
    socket.disconnect();
  };
}, [isRecording]);

服务器接收参考（因你已配置好转写逻辑，仅作示例）： #

服务器需监听audio事件，接收ArrayBuffer后转为Blob处理，转写完成后可通过Socket返回结果：

// Node.js服务器示例
socket.on("audio", async ({ roomId, data }) => {
  const audioBlob = new Blob([data], { type: "audio/webm" });
  // 调用转写服务处理音频
  const transcript = await transcribeAudio(audioBlob);
  // 发送转写结果到对应房间
  io.to(roomId).emit("transcript-result", transcript);
});

完整修改后的TranscriptComponent核心代码： #

import React, { useState, useEffect } from 'react';
// 导入你的Socket实例和其他依赖

export const TranscriptComponent = (props: TranscriptComponentProps) => {
  const [isRecording, setIsRecording] = useState(false);
  const [stream, setStream] = useState<MediaStream | null>(null);
  const [systemAudioStream, setSystemAudioStream] = useState<MediaStream | null>(null);
  const [timerInterval, setTimerInterval] = useState<NodeJS.Timeout | null>(null);
  const [transcriptLength, setTranscriptLength] = useState(0);

  // Socket连接状态监听
  useEffect(() => {
    socket.on("connect", () => console.log("Socket connected"));
    socket.on("disconnect", () => {
      console.log("Socket disconnected");
      if (isRecording) {
        alert("与服务器连接断开，已停止录制");
        stopRecording();
      }
    });
    socket.on("error", (err) => console.error("Socket error:", err));

    return () => {
      socket.off("connect");
      socket.off("disconnect");
      socket.off("error");
      socket.disconnect();
    };
  }, [isRecording]);

  const startRecording = (id: string) => {
    socket.connect();
    console.log("=recording started");

    Promise.all([
      navigator.mediaDevices.getUserMedia({ 
        audio: { echoCancellation: false, noiseSuppression: false }
      }),
      navigator.mediaDevices.getDisplayMedia({ audio: true, video: false })
    ]).then(([micStream, systemStream]) => {
      systemStream.getVideoTracks().forEach(track => track.stop());
      const combinedStream = new MediaStream();
      micStream.getAudioTracks().forEach(track => combinedStream.addTrack(track));
      systemStream.getAudioTracks().forEach(track => combinedStream.addTrack(track));

      const mimeTypes = ["audio/mp4", "audio/webm"].filter(type => MediaRecorder.isTypeSupported(type));
      if (mimeTypes.length === 0) return alert("Browser not supported");

      setIsRecording(true);
      setStream(combinedStream);
      setSystemAudioStream(systemStream);
      setTimerInterval(setInterval(() => setTranscriptLength(t => t + 1), 1000));

      const recorder = new MediaRecorder(combinedStream, { mimeType: mimeTypes[0] });
      recorder.addEventListener("dataavailable", async (event) => {
        if (event.data.size > 0 && socket.connected) {
          console.log("sending audio");
          const arrayBuffer = await event.data.arrayBuffer();
          socket.emit("audio", { roomId: props.roomId, data: arrayBuffer });
        } else {
          console.log("no data available or socket disconnected");
        }
      });
      recorder.start(1000);
    }).catch(err => {
      console.error("获取媒体流失败:", err);
      alert("无法获取麦克风或系统音频权限，请检查浏览器设置");
    });
  };

  const stopRecording = () => {
    stream?.getTracks().forEach(track => track.stop());
    systemAudioStream?.getTracks().forEach(track => track.stop());
    setIsRecording(false);
    clearInterval(timerInterval!);
    socket.emit("stop-transcript", { roomId: props.roomId });
    console.log("recording stopped");
    setStream(null);
    setSystemAudioStream(null);
    setTranscriptLength(0);
  };

  // ... 你的渲染逻辑
};

内容的提问来源于stack exchange，提问作者Abdul Waris