获取扬声器输出MediaStream用于微软认知服务实时语音转写遇问题

问题背景 #

需要捕获扬声器输出的MediaStream，传输至微软认知服务进行实时语音转写。核心难点在于获取扬声器输出流，只要能将其转换为MediaStream，即可通过AudioConfig.fromStreamInput(myMediaStream)配置语音转写的输入源。

现有实现的错误 #

参考相关方案时，执行speaker.addTrack(stream.getAudioTracks()[0].clone())出现错误：Cannot read properties of undefined (reading 'clone')，具体实现代码如下：

const getAudioStream = async () => {
    const speaker = new MediaStream;
    if (navigator.getDisplayMedia) {
        navigator.getDisplayMedia({
            video: true ,
            audio: true
        }).then(stream => {
            speaker.addTrack(stream.getAudioTracks()[0].clone());
            // stopping and removing the video track to enhance the performance
            stream.getVideoTracks()[0].stop();
            stream.removeTrack(stream.getVideoTracks()[0]);
        }).catch((error) => {
            console.log(error);
        });
    } else if (navigator.mediaDevices.getDisplayMedia) {
        navigator.mediaDevices.getDisplayMedia({
            video: true ,
            audio: true
        }).then(stream => {
            speaker.addTrack(stream.getAudioTracks()[0].clone());
            // stopping and removing the video track to enhance the performance
            stream.getVideoTracks()[0].stop();
            stream.removeTrack(stream.getVideoTracks()[0]);
        }).catch((error) => {
            console.log(error);
        });
    }
   return speaker;
}

useEffect(() => { const startStreaming = async () => { const speaker = await getAudioStream(); const audioConfig = AudioConfig.fromStreamInput(speaker); speechrecognizer.audioConfig = audioConfig; } startStreaming(); }, []);

过往相关实现 #

曾实现过同时录制麦克风和扬声器流的React Hook，但用类似方法仅能获取麦克风流，无法捕获扬声器流，代码如下：

import { useEffect, useRef, useState } from 'react';

export const useAudioRecorder = () => { 
    const [isRecording, setIsRecording] = useState(false); 
    const [audioBlob, setAudioBlob] = useState(null); 
    const mediaRecorder = useRef(null); 
    const audioStream = useRef(null);

    const startRecording = async () => { 
        try { 
            const microphoneStream = await navigator.mediaDevices.getUserMedia({ audio: true }); 
            const speakerStream = await navigator.mediaDevices.getUserMedia({ audio: { echoCancellation: false } });
            const audioContext = new AudioContext();
            const microphoneSource = audioContext.createMediaStreamSource(microphoneStream);
            const speakerSource = audioContext.createMediaStreamSource(speakerStream);
            const mixedOutput = audioContext.createMediaStreamDestination();

            microphoneSource.connect(mixedOutput);
            speakerSource.connect(mixedOutput);

            audioStream.current = mixedOutput.stream;
            mediaRecorder.current = new MediaRecorder(audioStream.current);
            
            mediaRecorder.current.ondataavailable = handleDataAvailable;
            mediaRecorder.current.start();
            
            setIsRecording(true);
        } catch (error) {
            console.error('Error starting recording:', error);
        }
    };

    const stopRecording = () => { 
        if (mediaRecorder.current && isRecording) { 
            mediaRecorder.current.stop(); 
            setIsRecording(false); 
        } 
    };

    const handleDataAvailable = (event) => { 
        const audioBlob = new Blob([event.data], { type: 'audio/wav' }); 
        setAudioBlob(audioBlob); 
    };

    useEffect(() => () => { 
        // Clean up the streams when the component unmounts 
        if (audioStream.current) { 
            audioStream.current.getTracks().forEach(track => track.stop()); 
        } 
    }, []);

    return { isRecording, audioBlob, startRecording, stopRecording }; 
};

解决方案 #

错误原因分析 #

1. 异步逻辑问题：原getAudioStream函数使用then回调处理异步操作，直接返回空的speaker对象时，异步流捕获尚未完成，导致后续操作无效。
2. 音频轨道检查缺失：若用户在共享弹窗未选择包含系统音频，stream.getAudioTracks()会返回空数组，取[0]得到undefined，调用clone()就会报错。
3. 过往Hook的误区：navigator.mediaDevices.getUserMedia只能访问输入设备（麦克风），无法直接捕获扬声器输出的音频，必须使用getDisplayMedia来获取系统音频流。

修正后的代码实现 #

const getAudioStream = async () => {
    let speakerStream = new MediaStream();
    try {
        // 统一使用现代浏览器支持的navigator.mediaDevices.getDisplayMedia
        const stream = await navigator.mediaDevices.getDisplayMedia({
            video: false, // 无需视频，直接关闭以提升性能
            audio: {
                echoCancellation: false,
                noiseSuppression: false,
                autoGainControl: false
            }
        });

        const audioTracks = stream.getAudioTracks();
        if (audioTracks.length > 0) {
            speakerStream.addTrack(audioTracks[0].clone());
            // 停止原流的所有轨道，避免资源占用
            stream.getTracks().forEach(track => track.stop());
        } else {
            throw new Error("未捕获到系统音频轨道，请确保共享时选择包含系统音频");
        }
    } catch (error) {
        console.error("获取扬声器流失败:", error);
        throw error; // 抛出错误让调用方处理
    }
    return speakerStream;
};

// 修正后的初始化逻辑
useEffect(() => {
    const startStreaming = async () => {
        try {
            const speaker = await getAudioStream();
            const audioConfig = AudioConfig.fromStreamInput(speaker);
            speechrecognizer.audioConfig = audioConfig;
        } catch (error) {
            console.error("初始化语音识别失败:", error);
        }
    };
    startStreaming();
}, []);

关键注意事项 #

• 用户在浏览器弹出的共享窗口中，必须勾选包含系统音频选项（不同浏览器名称略有差异，如Chrome为“分享音频”，Edge为“包括系统音频”），否则无法捕获扬声器输出。
• 若需要同时捕获麦克风和扬声器流，可以在获取扬声器流后，通过getUserMedia获取麦克风流，再用AudioContext将两个流混合，生成新的MediaStream。

内容的提问来源于stack exchange，提问作者Abubakar Bincom