移动端MediaRecorder结合AudioContext录制时事件触发异常

我在Angular 15/Ionic 6环境下实现了录制经Canvas处理的视频流，并结合原视频的AudioContext音频流合成录制文件。桌面端完全正常，但移动端（Safari、Chrome）添加AudioContext时成功率仅约50%，移除AudioContext则100%正常。

核心逻辑是：将Canvas捕获的视频轨道与AudioContext获取的音频轨道合并为MediaStream，再用MediaRecorder录制；启动录制后逐帧处理视频到Canvas，视频结束时调用MediaRecorder.stop()。

相关代码 #

HTML #

<ion-button (click)="startVideoProcessing()">Process Video</ion-button>
<video id="uploadedVideo" #uploadedVideo autoplay webkit-playsinline playsinline></video>
<canvas id="canvasVideo" #canvasVideo></canvas>
<canvas id="segmentationMaskCanvasVideo" #segmentationMaskCanvasVideo></canvas>

MediaRecorder核心代码 #

async videoProcessing() {
    // audioContext: AudioContext;
    // sourceNode: MediaElementAudioSourceNode;
    this.stream = new MediaStream(
      [...this.videoCanvas.captureStream(30).getVideoTracks(),
        this.getAudioTrack()
      ]);
    const options = {
        audioBitsPerSecond: 128000,
        videoBitsPerSecond: 2500000,
        mimeType: this.mimeType,
    };
    // recorder: MediaRecorder
    this.recorder = new MediaRecorder(this.stream, options);
    this.recorder.start();
    this.recorder.ondataavailable = (event) => {
        if (event.data && event.data.size > 0) {
            this.chunks.push(event.data);
        }
    };
    this.recorder.onerror = (event) => {
      console.error(event);
      throw new Error(event.toString());
    };
    this.recorder.onstop = () => {
      const videoBlob = new Blob(this.chunks, { type: this.mimeType });
      const videoFile = new File([videoBlob], this.mimeType === 'video/mp4' ? 'video-green-screen.mp4' : 'video-green-screen.webm', { type: this.mimeType });
      this.fileSelected.emit(videoFile);
    };
    this.registerVideoFrameCallback();
  }

  getAudioTrack() {
    if ((this.video as any).captureStream) {
      // videoAudioTrack: MediaStreamTrack 
      this.videoAudioTrack = (this.video as any).captureStream().getAudioTracks()[0];
    } else if ((this.video as any).mozCaptureStream) {
      this.videoAudioTrack = (this.video as any).mozCaptureStream().getAudioTracks()[0];
    } else {
      const destination = this.audioContext.createMediaStreamDestination();
      this.sourceNode.connect(destination);
      this.videoAudioTrack = destination.stream.getAudioTracks()[0];
    }
    return this.videoAudioTrack;
  }

其他辅助TS代码 #

initVideoElements() {
    this.video = this.videoElementRef.nativeElement;
    this.video.style.display = 'none';
    this.videoCanvas = this.videoCanvasElementRef.nativeElement;
    this.videoCanvas.style.display = 'none';
    this.videoCanvasCtx = this.videoCanvas.getContext('2d');
    this.segmentationMaskCanvas = this.segmentationMaskCanvasElementRef.nativeElement;
    this.segmentationMaskCanvasCtx = this.segmentationMaskCanvas.getContext('2d');
    this.segmentationMaskCanvas.style.display = 'none';
  }

  onVideoFileSelected(file: File) {
    this.videoFile = file; // videoFile: File;
  }

  async startVideoProcessing() {
    // @ViewChild('uploadedVideo', { static: false }) videoElementRef: ElementRef<HTMLVideoElement>;
    // video: HTMLVideoElement;
    this.video.src = URL.createObjectURL(this.videoFile);
    this.silenceVideoAudio();
    this.video.onloadeddata = async () => {
      await this.videoProcessing();
    };
    this.video.onended = async () => {
      this.recorder.stop()
    };
  }

  // 静音视频音频（因为视频被隐藏）
  silenceVideoAudio() {
    this.audioContext = new AudioContext();
    this.sourceNode = this.audioContext.createMediaElementSource(this.video);
    const gainNode = this.audioContext.createGain();
    gainNode.gain.value = 0;
    this.sourceNode.connect(gainNode);
    gainNode.connect(this.audioContext.destination);
  }

  registerVideoFrameCallback(): void {
      const doSomethingWithTheFrame = (now: number, metadata: any) => {
        this.processingProgress = parseFloat(metadata.mediaTime.toFixed(2)) / this.video.duration;
        this.cdRef.detectChanges();
        this.imageSegmenter.segmentForVideo(this.video, metadata.presentedFrames, async segmentationMask => await this.segmentationVideoCallback(segmentationMask));
      this.videoElementRef.nativeElement.requestVideoFrameCallback(doSomethingWithTheFrame);
      };
      this.video.currentTime = 0;
      this.videoElementRef.nativeElement.requestVideoFrameCallback(doSomethingWithTheFrame);
  }

  async segmentationVideoCallback(segmentationMask){
    // 将分割结果绘制到Canvas，用于捕获视频流
    await this.drawSegmentationResult(segmentationMask.categoryMask);
  }

问题现象 #

• 视频触发onended时，MediaRecorder已处于inactive状态，但移动端仅部分情况触发ondataavailable和onstop事件
• 无报错信息，但锁屏再返回后事件可能触发
• 尝试过requestData()、runOutsideAngular、timeslice、停止轨道再调用stop()等方案，均未解决稳定性问题

解决方案 #

方案1：优化AudioContext与MediaStream的同步逻辑 #

移动端浏览器对AudioContext的状态管理更严格，需避免过早或异步创建音频轨道：

1. 确保AudioContext在用户交互（点击按钮）时初始化，符合浏览器自动播放策略
2. 在videoProcessing中延迟创建音频轨道，确保视频完全加载：

async videoProcessing() {
    // 先启动Canvas视频流
    const canvasStream = this.videoCanvas.captureStream(30);
    // 等待视频音频轨道准备完成
    await new Promise(resolve => {
        if (this.video.readyState >= 2) resolve(true);
        else this.video.oncanplaythrough = resolve;
    });
    // 获取音频轨道并合并到流中
    const audioTrack = this.getAudioTrack();
    this.stream = new MediaStream([...canvasStream.getVideoTracks(), audioTrack]);
    
    // 后续MediaRecorder初始化逻辑不变
}

3. 修改silenceVideoAudio，确保AudioContext在用户点击时激活：

async startVideoProcessing() {
    this.video.src = URL.createObjectURL(this.videoFile);
    // 等待用户交互激活AudioContext
    await this.silenceVideoAudio();
    this.video.onloadeddata = async () => {
      await this.videoProcessing();
    };
    // ...其他逻辑
}

async silenceVideoAudio() {
    this.audioContext = new AudioContext();
    // 等待AudioContext激活（部分移动端需要显式resume）
    if (this.audioContext.state === 'suspended') {
        await this.audioContext.resume();
    }
    this.sourceNode = this.audioContext.createMediaElementSource(this.video);
    const gainNode = this.audioContext.createGain();
    gainNode.gain.value = 0;
    this.sourceNode.connect(gainNode);
    gainNode.connect(this.audioContext.destination);
}

方案2：替换AudioContext为直接捕获视频音频轨道 #

优先使用video.captureStream()获取音频轨道，避免通过AudioContext中转，减少兼容性问题：

getAudioTrack() {
    // 优先使用原生captureStream获取音频，仅在不支持时才用AudioContext
    if ('captureStream' in this.video) {
        const videoStream = (this.video as any).captureStream();
        this.videoAudioTrack = videoStream.getAudioTracks()[0];
        // 静音原视频音频（无需AudioContext）
        this.video.muted = true;
    } else if ('mozCaptureStream' in this.video) {
        const videoStream = (this.video as any).mozCaptureStream();
        this.videoAudioTrack = videoStream.getAudioTracks()[0];
        this.video.muted = true;
    } else {
        // fallback到原AudioContext逻辑
        const destination = this.audioContext.createMediaStreamDestination();
        this.sourceNode.connect(destination);
        this.videoAudioTrack = destination.stream.getAudioTracks()[0];
    }
    return this.videoAudioTrack;
}

方案3：强制触发MediaRecorder数据回调 #

在视频结束时，先手动停止所有轨道，再延迟调用recorder.stop()，确保数据完整写入：

this.video.onended = async () => {
    // 先停止流中的所有轨道
    this.stream.getTracks().forEach(track => track.stop());
    // 延迟100ms调用stop，确保数据缓冲区被清空
    setTimeout(() => {
        if (this.recorder.state !== 'inactive') {
            this.recorder.stop();
        }
        // 手动触发ondataavailable（如果未自动触发）
        if (this.chunks.length === 0 && this.stream) {
            this.recorder.ondataavailable({ data: new Blob([], { type: this.mimeType }) });
            this.recorder.onstop();
        }
    }, 100);
};

方案4：使用WebCodecs API替代MediaRecorder（进阶） #

如果MediaRecorder兼容性问题无法解决，可以尝试WebCodecs直接编码视频帧和音频数据，更精细控制录制流程：

• 从Canvas获取视频帧，从AudioContext获取音频样本
• 使用VideoEncoder和AudioEncoder分别编码
• 合并编码后的数据包生成最终文件

这种方案兼容性稍差，但稳定性更高，适合对录制质量有严格要求的场景。

内容的提问来源于stack exchange，提问作者Eric Belisle Giddings