如何利用Screen Capture API实现主副显示器合并截图与推流？

Great question! Let's break down how to expand your single-screen capture setup to handle multiple monitors—whether you want to capture the entire combined desktop in one go, or manually merge individual screen captures for more control.

Option 1: Capture the Entire Combined Desktop (Simplest Approach) #

Modern browsers let users select their "Entire Screen" when using getDisplayMedia, which automatically captures all connected monitors as a single, merged video stream. This is the most efficient way for both screenshots and streaming.

Updated Screenshot Code #

Modify your existing function to support full desktop capture (users will see an option to select "Entire Screen" when prompted):

const screenshotMultiScreen = async() => { 
  const mediaDevices = navigator.mediaDevices as any; 
  // Adjust options to let users choose entire screen or individual displays
  const displayMediaOptions = { 
    video: { 
      mediaSource: 'screen',
      // Optional: Force capture of entire desktop (some browsers auto-include this option)
      // displaySurface: 'monitor' // Use this if you want users to pick individual monitors
    },
    audio: false // Omit if you need system audio
  }; 

  const stream = await mediaDevices.getDisplayMedia(displayMediaOptions); 
  const track = stream.getVideoTracks()[0]; 

  const imageCapture = new ImageCapture(track); 
  const bitmap = await imageCapture.grabFrame(); 

  // Clean up resources
  track.stop();
  stream.getTracks().forEach(t => t.stop());

  // Draw merged frame to canvas (same as your original logic)
  const canvas = document.getElementById('fake') || document.createElement('canvas');
  canvas.width = bitmap.width;
  canvas.height = bitmap.height;
  const context = canvas.getContext('2d');
  context.drawImage(bitmap, 0, 0, bitmap.width, bitmap.height);

  // Convert to File object
  const image = canvas.toDataURL();
  const res = await fetch(image);
  const buff = await res.arrayBuffer();
  const file = new File([buff], `multi-screen-photo_${new Date()}.jpg`, { type: 'image/jpeg' });
  
  return [file]; 
};

Streaming with the Combined Desktop Stream #

To stream the merged display, use the same stream directly with MediaRecorder or an RTCPeerConnection (for live streaming):

const startMultiScreenStream = async() => {
  const mediaDevices = navigator.mediaDevices as any;
  const displayMediaOptions = {
    video: { mediaSource: 'screen' },
    audio: true // Include if you need system audio
  };
  const stream = await mediaDevices.getDisplayMedia(displayMediaOptions);

  // Example: Record the stream to a file
  const recorder = new MediaRecorder(stream, { mimeType: 'video/webm' });
  const chunks = [];

  recorder.ondataavailable = (e) => chunks.push(e.data);
  recorder.onstop = () => {
    const blob = new Blob(chunks, { type: 'video/webm' });
    // Download or send the blob to your streaming server
    const downloadLink = document.createElement('a');
    downloadLink.href = URL.createObjectURL(blob);
    downloadLink.download = `multi-screen-stream_${new Date()}.webm`;
    downloadLink.click();
  };

  recorder.start();
  // Stop recording after 30 seconds (adjust as needed)
  setTimeout(() => recorder.stop(), 30000);
};

Option 2: Manually Capture & Merge Individual Screens #

If you need to process each screen separately before merging (e.g., add overlays to specific monitors), you can capture each screen individually and combine them on a single canvas.

Step 1: Capture Individual Screens #

First, create a helper to capture a single monitor and get its position/size (browser exposes this via track settings):

async function captureSingleMonitor() {
  const mediaDevices = navigator.mediaDevices as any;
  const displayMediaOptions = {
    video: {
      mediaSource: 'screen',
      displaySurface: 'monitor' // Force users to select a single monitor
    },
    audio: false
  };
  const stream = await mediaDevices.getDisplayMedia(displayMediaOptions);
  const track = stream.getVideoTracks()[0];
  const settings = track.getSettings(); // Contains x/y position and width/height of the monitor

  return { stream, track, settings };
}

Step 2: Merge Screens onto a Single Canvas #

Loop to capture multiple screens, calculate the combined canvas size, then draw each screen's frame in its correct position:

async function mergeScreenshots() {
  const captures = [];
  let continueCapturing = true;

  // Let users select monitors until they cancel
  while (continueCapturing) {
    try {
      const capture = await captureSingleMonitor();
      captures.push(capture);
    } catch (err) {
      // User canceled selection, stop capturing
      continueCapturing = false;
    }
  }

  if (captures.length === 0) throw new Error('No monitors were captured');

  // Calculate the size of the merged canvas (covers all monitors)
  let maxCanvasWidth = 0;
  let maxCanvasHeight = 0;
  captures.forEach(({ settings }) => {
    const totalWidth = settings.x + settings.width;
    const totalHeight = settings.y + settings.height;
    if (totalWidth > maxCanvasWidth) maxCanvasWidth = totalWidth;
    if (totalHeight > maxCanvasHeight) maxCanvasHeight = totalHeight;
  });

  // Create the merged canvas
  const mergedCanvas = document.createElement('canvas');
  mergedCanvas.width = maxCanvasWidth;
  mergedCanvas.height = maxCanvasHeight;
  const ctx = mergedCanvas.getContext('2d');

  // Draw each monitor's frame to the merged canvas
  for (const { track, settings } of captures) {
    const imageCapture = new ImageCapture(track);
    const bitmap = await imageCapture.grabFrame();
    // Draw the frame at the monitor's actual position in the desktop layout
    ctx.drawImage(bitmap, settings.x, settings.y, bitmap.width, bitmap.height);
    // Clean up resources
    track.stop();
    captures.stream.getTracks().forEach(t => t.stop());
  }

  // Convert merged canvas to File (same as your original logic)
  const image = mergedCanvas.toDataURL();
  const res = await fetch(image);
  const buff = await res.arrayBuffer();
  const file = new File([buff], `merged-screens-photo_${new Date()}.jpg`, { type: 'image/jpeg' });
  
  return [file];
}

Streaming Merged Screens #

To stream the manually merged canvas, convert it to a media stream with captureStream():

// After creating mergedCanvas
const canvasStream = mergedCanvas.captureStream(30); // 30 FPS
// Use this stream with MediaRecorder or RTCPeerConnection for streaming
const recorder = new MediaRecorder(canvasStream);
// ... same recording logic as Option 1

Key Notes #

• User Experience: Option 1 is more user-friendly (one permission prompt) vs. Option 2 (multiple prompts for each monitor).
• Performance: Option 1 is far more efficient, as the browser handles merging at the hardware level. Manual merging can cause lag with high-resolution monitors.
• Browser Support: getDisplayMedia and ImageCapture are supported in Chrome, Edge, and Firefox. Safari has limited support for some features (e.g., displaySurface).
• Resource Cleanup: Always stop tracks after capture to avoid memory leaks—this is crucial when handling multiple streams.

内容的提问来源于stack exchange，提问作者Manoj Joshi