如何在NodeJS中高帧率录制并处理屏幕捕获画面？

Great question—capturing 60fps screen footage directly into Node.js without writing to disk or dealing with lag is definitely achievable, but you’ll need to bypass the common pitfalls of libraries like robotjs or RTMP’s built-in buffering. Let’s walk through the most reliable solutions:

1. Use FFmpeg with Pipe Output (Best Cross-Platform Option) #

FFmpeg is optimized for high-performance media capture and can stream raw frame data directly to Node.js via stdout—no disk writes required, and it easily hits 60fps if your hardware supports it.

How to Set It Up: #

First, ensure FFmpeg is installed on your system (or use the ffmpeg-static npm package to bundle it with your app). Then spawn an FFmpeg child process that captures the screen and outputs raw video frames to a pipe:

const { spawn } = require('child_process');
// Use ffmpeg-static if you don't want to rely on system FFmpeg
// const ffmpegPath = require('ffmpeg-static').path;

// Adjust capture parameters based on your OS:
// - Windows: use "gdigrab" as the input format
// - Mac: use "avfoundation" (add "-i 1" to capture main screen)
// - Linux: use "x11grab" (add "-i :0.0" for primary display)
const ffmpegProcess = spawn('ffmpeg', [
  '-f', 'gdigrab',
  '-framerate', '60',
  '-i', 'desktop',
  '-pix_fmt', 'bgr24', // Raw pixel format (easy to parse in Node)
  '-f', 'rawvideo',
  'pipe:1' // Send output to stdout
]);

// Replace with your screen resolution
const SCREEN_WIDTH = 1920;
const SCREEN_HEIGHT = 1080;
const FRAME_SIZE = SCREEN_WIDTH * SCREEN_HEIGHT * 3; // 3 bytes per pixel (BGR)

// Handle raw frame data from FFmpeg
ffmpegProcess.stdout.on('data', (rawData) => {
  // Process full frames (avoid partial frame data)
  let offset = 0;
  while (offset + FRAME_SIZE <= rawData.length) {
    const frameBuffer = rawData.slice(offset, offset + FRAME_SIZE);
    // Pass the frame to your processing function
    processScreenFrame(frameBuffer, SCREEN_WIDTH, SCREEN_HEIGHT);
    offset += FRAME_SIZE;
  }
});

// Example processing function (customize this!)
function processScreenFrame(buffer, width, height) {
  // Convert BGR buffer to RGBA if needed (common for image processing libraries)
  const rgbaBuffer = Buffer.alloc(width * height * 4);
  for (let i = 0; i < buffer.length; i += 3) {
    rgbaBuffer[i] = buffer[i + 2]; // Red
    rgbaBuffer[i + 1] = buffer[i + 1]; // Green
    rgbaBuffer[i + 2] = buffer[i]; // Blue
    rgbaBuffer[i + 3] = 255; // Alpha (opaque)
  }

  // Add your logic here: object detection, pixel analysis, etc.
  console.log(`Processed frame: ${Date.now()}`);
}

Key Tips: #

• Use ffmpeg-static to avoid requiring users to install FFmpeg manually.
• Run frame processing in a worker thread to avoid blocking the Node.js event loop (critical for maintaining 60fps).
• Adjust the capture region with -i desktop -offset_x X -offset_y Y -video_size WxH if you don’t need the entire screen (smaller regions = faster processing).

2. Electron Desktop Capturer (If Using Electron) #

If your app is built on Electron, the built-in desktopCapturer API leverages Chromium’s high-performance media stack to capture screen frames at 60fps with minimal lag. It gives you direct access to a MediaStream, which you can feed into a canvas to extract frames.

Example Code (Render Process): #

const { desktopCapturer } = require('electron');
const { ipcRenderer } = require('electron');

async function startHighFpsCapture() {
  // Get screen capture sources
  const sources = await desktopCapturer.getSources({ types: ['screen'] });
  const mainScreenSource = sources.find(source => source.name.includes('Screen'));

  // Create a MediaStream from the screen
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: false,
    video: {
      mandatory: {
        chromeMediaSource: 'desktop',
        chromeMediaSourceId: mainScreenSource.id,
        minFrameRate: 60,
        maxFrameRate: 60,
        width: 1920,
        height: 1080
      }
    }
  });

  // Load stream into a hidden video element
  const video = document.createElement('video');
  video.srcObject = stream;
  video.play();

  // Capture frames with requestAnimationFrame (syncs with display refresh rate)
  const canvas = document.createElement('canvas');
  const ctx = canvas.getContext('2d');
  
  function captureAndProcessFrame() {
    canvas.width = video.videoWidth;
    canvas.height = video.videoHeight;
    ctx.drawImage(video, 0, 0);
    
    // Get raw pixel data
    const imageData = ctx.getImageData(0, 0, canvas.width, canvas.height);
    
    // Send data to main process (or process directly in render)
    ipcRenderer.send('process-screen-frame', imageData);
    
    requestAnimationFrame(captureAndProcessFrame);
  }

  requestAnimationFrame(captureAndProcessFrame);
}

startHighFpsCapture();

3. Why Your Previous Approaches Failed (& Quick Fixes If You Want to Revisit Them) #

• robotjs: It’s not optimized for high-frame-rate capture because it uses synchronous screen reads that block the event loop. If you must use it, try capturing only a small region and offloading processing to worker threads—but it’s unlikely to hit 60fps on most systems.
• RTMP Server + OBS: RTMP introduces intentional buffering (1–3 seconds) to handle network instability, which makes it unsuitable for real-time processing. You could try reducing OBS’s buffer size to the minimum, but it’s still not as efficient as direct pipe/streaming.

内容的提问来源于stack exchange，提问作者Kirk