如何降低iPhone摄像头经FFmpeg推流至v4l2loopback的延迟？

我正在尝试将iPhone摄像头画面通过局域网推流到电脑，最终输出到v4l2loopback设备，但画面延迟至少1.5秒，完全无法满足项目需求。以下是我的实现细节和排查过程：

1. 前端推流实现（iPhone端） #

通过浏览器MediaRecorder捕获摄像头画面，用WebSocket将MP4分片发送到服务器。虽然WebRTC可能是更低延迟的方案，但当前网络延迟暂时在可接受范围内。

async function beginCameraStream() {
  const mediaStream = await navigator.mediaDevices.getUserMedia({
    video: { facingMode: "user" },
  });

  websocket = new WebSocket(SERVER_URL);

  websocket.onopen = () => {
    console.log("WS connected");

    const options = { mimeType: "video/mp4", videoBitsPerSecond: 1_000_000 };
    mediaRecorder = new MediaRecorder(mediaStream, options);

    mediaRecorder.ondataavailable = async (event) => {
      // 为了测量延迟，在视频分片前添加时间戳
      const timestamp = Date.now();
      const timestampBuffer = new ArrayBuffer(8);
      const dataView = new DataView(timestampBuffer);
      dataView.setBigUint64(0, BigInt(timestamp), true);
      const data = await event.data.bytes();

      const result = new Uint8Array(data.byteLength + 8);
      result.set(new Uint8Array(timestampBuffer), 0);
      result.set(data, 8);

      websocket.send(result);
    };

    mediaRecorder.start(100); // 每100ms生成一个分片
  };
}

2. 后端处理逻辑（电脑端） #

用Bun搭建WebSocket服务器，接收分片后通过自定义可读流传递给FFmpeg，转码输出到v4l2loopback设备：

import { serve } from "bun";
import { Readable } from "stream";

const V4L2LOOPBACK_DEVICE = "/dev/video10";

export const setupFFmpeg = (v4l2device) => {
  // prettier-ignore
  return spawn("ffmpeg", [
    '-i', 'pipe:0',           // 从标准输入读取
    '-pix_fmt', 'yuv420p',    // 像素格式
    '-r', '30',               // 目标帧率30fps
    '-f', 'v4l2',             // 输出格式
    v4l2device, // 输出到v4l2loopback设备
  ]);
};

export class FfmpegStream extends Readable {
  _read() {
    // 当流需要更多数据时触发，我们在收到分片时推送数据
  }
}

function main() {
  const ffmpeg = setupFFmpeg(V4L2LOOPBACK_DEVICE);
  serve({
    port: 8000,
    fetch(req, server) {
      if (server.upgrade(req)) {
        return; // 升级为WebSocket连接
      }
    },
    websocket: {
      open(ws) {
        console.log("客户端已连接");
        const stream = new FfmpegStream();
        stream.pipe(ffmpeg?.stdin);

        ws.data = {
          stream,
          received: 0,
        };
      },
      async message(ws, message) {
        const view = new DataView(message.buffer, 0, 8);
        const ts = Number(view.getBigUint64(0, true));
        ws.data.received += message.byteLength;
        const chunk = new Uint8Array(message.buffer, 8, message.byteLength - 8);

        ws.data.stream.push(chunk);

        console.log(
          [
            `latency: ${Date.now() - ts} ms`,
            `chunk: ${message.byteLength}`,
            `total: ${ws.data.received}`,
          ].join(" | "),
        );
      },
    },
  });
}

main();

测试方式 #

通过以下命令查看v4l2loopback设备的画面：

cvlc v4l2:///dev/video10

延迟排查分析 #

网络延迟排除 #

从日志看，网络传输延迟在140-200ms左右，虽偏高但远低于1.5s的总延迟：

latency: 140 ms | chunk: 661 Bytes | total: 661 Bytes
latency: 206 ms | chunk: 16.76 KB | total: 17.41 KB
latency: 141 ms | chunk: 11.28 KB | total: 28.68 KB
latency: 141 ms | chunk: 13.05 KB | total: 41.74 KB
latency: 199 ms | chunk: 11.39 KB | total: 53.13 KB
latency: 141 ms | chunk: 16.94 KB | total: 70.07 KB
latency: 139 ms | chunk: 12.67 KB | total: 82.74 KB
latency: 142 ms | chunk: 13.14 KB | total: 95.88 KB

局域网传输15KB数据延迟约150ms确实偏高，可能与路由器有关，但不是主要问题

FFmpeg吞吐量排除 #

FFmpeg日志显示处理速度能跟上，帧率和转码效率正常：

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'pipe:0':
  Metadata:
    major_brand     : iso5
    minor_version   : 1
    compatible_brands: isomiso5hlsf
    creation_time   : 2025-03-09T17:16:49.000000Z
  Duration: 00:00:01.38, start:
0.000000, bitrate: N/A
    Stream #0:0(und): Video: h264 (Baseline) (avc1 / 0x31637661), yuvj420p(pc), 1280x720, 4012 kb/s, 57.14 fps, 29.83 tbr, 600 tbn, 1200 tbc (default)
    Metadata:
      rotate          : 90
      creation_time   : 2025-03-09T17:16:49.000000Z
      handler_name    : Core Media Video
    Side data:
      displaymatrix: rotation of -90.00 degrees

Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> rawvideo (native))

[swscaler @ 0x55d8d0b83100] deprecated pixel format used, make sure you did set range correctly

Output #0, video4linux2,v4l2, to '/dev/video10':
  Metadata:
    major_brand     : iso5
    minor_version   : 1
    compatible_brands: isomiso5hlsf
    encoder         : Lavf58.45.100

Stream #0:0(und): Video: rawvideo (I420 / 0x30323449), yuv420p, 720x1280, q=2-31, 663552 kb/s, 60 fps, 60 tbn, 60 tbc (default)
    Metadata:
      encoder         : Lavc58.91.100 rawvideo
      creation_time   : 2025-03-09T17:16:49.000000Z
      handler_name    : Core Media Video
    Side data:
      displaymatrix: rotation of -0.00 degrees

frame=   99 fps=0.0 q=-0.0 size=N/A time=00:00:01.65 bitrate=N/A dup=50 drop=0 speed=2.77x
frame=  137 fps=114 q=-0.0 size=N/A time=00:00:02.28 bitrate=N/A dup=69 drop=0 speed=1.89x
frame=  173 fps= 98 q=-0.0 size=N/A time=00:00:02.88 bitrate=N/A dup=87 drop=0 speed=1.63x
frame=  210 fps= 86 q=-0.0 size=N/A time=00:00:03.50 bitrate=N/A dup=105 drop=0 speed=1.44x
frame=  249 fps= 81 q=-0.0 size=N/A time=00:00:04.15 bitrate=N/A dup=125 drop=0 speed=1.36
frame=  279 fps= 78 q=-0.0 size=N/A time=00:00:04.65 bitrate=N/A dup=139 drop=0 speed=1.31x

其他测试 #

• 将视频流直接写入video.mp4文件，用vlc打开同样需要等待约1.5秒才能播放
• 替换为OBS的v4l2输入源，延迟情况一致

更新：本地文件推流对比 #

用本地MP4文件通过管道推给FFmpeg时，几乎能立即播放，仅FFmpeg启动有0.2秒延迟：

cat video.mp4 | ffmpeg -re -i pipe:0 -pix_fmt yuv420p -f v4l2 /dev/video10 & ; sleep 0.2 && cvlc v4l2:///dev/video10

由此可以确定，问题出在实时推流的流程中，而非FFmpeg或v4l2loopback本身。

内容的提问来源于stack exchange，提问作者Made by Moses