如何结合WebWorkers与@mediapipe/tasks-vision解决姿态检测延迟同步问题

我使用Next.js 14.0.4和@mediapipe/tasks-vision@0.10.10，尝试通过WebWorker逐帧检测人体姿态，再在主线程中展示检测结果，但遇到了明显的延迟/同步问题——检测出的姿态标记和视频画面不同步，滞后于实际动作。

现有实现代码 #

pose-landmarker-worker.ts

import { PoseLandmarker } from "@mediapipe/tasks-vision";
import { createPoseLandmarker } from "@/lib/create-pose-landmarker";

let poseLandmarker: PoseLandmarker;

self.onmessage = async function (event: MessageEvent) {
    const data = event.data;

    if (data.action === "init") {
        if (!data.runningMode) {
            throw new Error("Could not initialize. The runningMode prop is not specified");
        }

        if (!poseLandmarker) {
            const poseLandmarkerInstance = await createPoseLandmarker(data.runningMode);

            if (!poseLandmarkerInstance) {
                throw new Error("Failed to create pose landmarker instance");
            }

            poseLandmarker = poseLandmarkerInstance;
            console.log("Created new pose landmarker instance");
        } else {
            console.warn("pose landmarker already initialized");
        }
    } else if (data.action === "detectForVideo") {
        const imageBitmap = data.frame as ImageBitmap;
        const result = poseLandmarker.detectForVideo(imageBitmap, data.timestamp);
        self.postMessage(result);
    }
};

page.tsx

"use client";
import React, { useEffect, useRef } from "react";
import { PoseLandmarkerResult } from "@mediapipe/tasks-vision";
import { drawConnectors } from "@/lib/draw-utils";

export default function WebWorkers2() {
    const videoRef = useRef<HTMLVideoElement>(null);

    const offscreenCanvasRef = useRef<OffscreenCanvas | null>(null);
    const offscreenCanvasCtxRef = useRef<OffscreenCanvasRenderingContext2D | null>(null);

    const drawCanvasRef = useRef<HTMLCanvasElement>(null);
    const workerRef = useRef<Worker | null>(null);

    const drawVideoToCanvas = () => {
        const video = videoRef.current;
        const worker = workerRef.current;

        if (!video || !worker) {
            console.log("invalid data");
            return;
        }
        const width = video.width;
        const height = video.height;

        if (!offscreenCanvasRef.current) {
            const offscreenCanvasInstance = new OffscreenCanvas(width, height);
            offscreenCanvasRef.current = offscreenCanvasInstance;
            offscreenCanvasCtxRef.current = offscreenCanvasInstance.getContext("2d", { willReadFrequently: true });
        }

        const ctx = offscreenCanvasCtxRef.current;
        const offscreenCanvas = offscreenCanvasRef.current;

        if (!offscreenCanvas || !ctx) {
            return;
        }

        ctx.drawImage(video, 0, 0, width, height);
        const imageBitmap = offscreenCanvas.transferToImageBitmap();
        const timestamp = performance.now();
        worker.postMessage({ action: "detectForVideo", frame: imageBitmap, timestamp }, [imageBitmap]);
        requestAnimationFrame(drawVideoToCanvas);
    };

    useEffect(() => {
        workerRef.current = new Worker(new URL("@/lib/pose-landmarker-worker.ts", import.meta.url));

        const ctx = drawCanvasRef.current?.getContext("2d")!;

        workerRef.current.onmessage = function (event: MessageEvent) {
            const result = event.data as PoseLandmarkerResult;

            if (ctx) {
                ctx.save();
                ctx.clearRect(0, 0, 360, 640);
                drawConnectors(result, ctx);
                ctx.restore();
            }
        };

        return () => {
            const worker = workerRef.current;
            if (worker) {
                worker.terminate();
            }
        };
    }, []);

    return (
        <div className="flex h-screen flex-col">
            <div className="flex ">
                <div className="relative">
                    <video
                        ref={videoRef}
                        onLoadedData={(event) => {
                            workerRef.current?.postMessage({
                                action: "init",
                                runningMode: "VIDEO",
                            });
                            setTimeout(drawVideoToCanvas, 3000);
                        }}
                        src="/static/videos/body_scan.mp4"
                        height={640}
                        width={360}
                        playsInline
                        autoPlay
                        muted
                        loop
                    />
                    <canvas ref={drawCanvasRef} className="absolute left-0 top-0" height={640} width={360} />
                </div>
            </div>
        </div>
    );
}

问题修复方案 #

1. 避免帧处理积压

WebWorker消息队列会被连续的帧请求塞满，导致旧帧的检测结果延迟返回。通过丢弃旧帧，只处理最新的一帧来解决：

修改pose-landmarker-worker.ts：

import { PoseLandmarker } from "@mediapipe/tasks-vision";
import { createPoseLandmarker } from "@/lib/create-pose-landmarker";

let poseLandmarker: PoseLandmarker;
let latestFrame: ImageBitmap | null = null;
let latestTimestamp: number = 0;

self.onmessage = async function (event: MessageEvent) {
    const data = event.data;

    if (data.action === "init") {
        if (!data.runningMode) {
            throw new Error("Could not initialize. The runningMode prop is not specified");
        }

        if (!poseLandmarker) {
            const poseLandmarkerInstance = await createPoseLandmarker(data.runningMode);

            if (!poseLandmarkerInstance) {
                throw new Error("Failed to create pose landmarker instance");
            }

            poseLandmarker = poseLandmarkerInstance;
            console.log("Created new pose landmarker instance");
        } else {
            console.warn("pose landmarker already initialized");
        }
    } else if (data.action === "detectForVideo") {
        // 替换旧帧，只保留最新的
        if (latestFrame) latestFrame.close();
        latestFrame = data.frame as ImageBitmap;
        latestTimestamp = data.timestamp;

        // 异步处理最新帧，不阻塞消息队列
        queueMicrotask(() => {
            if (!poseLandmarker || !latestFrame) return;
            const result = poseLandmarker.detectForVideo(latestFrame, latestTimestamp);
            self.postMessage(result);
            latestFrame.close();
            latestFrame = null;
        });
    }
};

2. 同步绘制时机

主线程收到检测结果后，要在浏览器的下一渲染周期绘制，确保和视频画面同步：

修改page.tsx中useEffect内的worker消息处理：

workerRef.current.onmessage = function (event: MessageEvent) {
    const result = event.data as PoseLandmarkerResult;

    // 用requestAnimationFrame确保在渲染帧绘制
    requestAnimationFrame(() => {
        if (ctx) {
            ctx.save();
            ctx.clearRect(0, 0, 360, 640);
            drawConnectors(result, ctx);
            ctx.restore();
        }
    });
};

3. 使用视频时间戳而非主线程时间

performance.now()是主线程的时间，和视频播放时间不同步，改用视频的currentTime转换为毫秒作为MediaPipe的时间戳：

修改page.tsx中的drawVideoToCanvas：

const timestamp = video.currentTime * 1000; // 转换为毫秒级时间戳
worker.postMessage({ action: "detectForVideo", frame: imageBitmap, timestamp }, [imageBitmap]);

4. 移除不必要的延迟

视频加载完成后无需延迟3秒启动帧捕获，直接调用drawVideoToCanvas：

onLoadedData={(event) => {
    workerRef.current?.postMessage({
        action: "init",
        runningMode: "VIDEO",
    });
    drawVideoToCanvas(); // 去掉setTimeout
}}

内容的提问来源于stack exchange，提问作者Christian Stamati