如何通过WebSocket将React视频流以Array Buffer发送至Python(FastAPI)？

我想通过WebSocket把用户的视频流以Array Buffer形式发送到Python后端。现在不知道怎么把getUserMedia获取的流转换成数组，再通过FastAPI的WebSocket发送过去。下面是我尝试的代码，求帮忙解决。

React 代码尝试 #

const userStream = useRef([]);

useEffect(() => {
  navigator.mediaDevices.getUserMedia({ audio: false, video: true }).then(stream => {
  userStream.current.srcobject = stream;

无法实现将userStream转换为数组对象的流程

var socket = new WebSocket('ws://localhost:8000/ws')
  socket.onopen = () => socket.send(userStream.current.srcobject)

  socket.onmessage = function (event) {
    console.log("[message] Data received from server:");
    if (event.data !== undefined && event.data !== null) {
      console.log(event.data)
    }
  }

  socket.onclose = () => {
    console.log("Connection Closed!");
  };
    
  socket.onerror = () => {
    console.log("WS Error");
  };

  return () => {
    socket.close();
  };
    });

}, []);

return (
    <div>
        <video controls style={{ height: 500, width: 500 }} autoPlay ref={userStream} />
    </div>
);

Python 代码尝试 #

无法处理从前端接收的数据

from fastapi import FastAPI, WebSocket  

@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket, data_image):
print("Accepting Connection")
await websocket.accept()
print('Accepted')
while True:
    try:

        image_bytes = await websocket.receive_bytes()
        print('Image Received')
        nparr = np.frombuffer(image_bytes, np.uint8)
        img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)

        # Do something with the image here
        # ...
        # Send a response back (optional)
        _, buffer = cv2.imencode('.jpg', img)
        response_bytes = buffer.tobytes()
        
        await websocket.send_bytes(response_bytes)
    except:
        pass
        break

前端（React）修改要点 #

1. 不能直接发送MediaStream对象：你现在直接把视频流对象传给socket.send()，WebSocket不支持这种类型，必须先把视频帧转成ArrayBuffer。
2. 用Canvas捕获帧：通过Canvas绘制视频的每一帧，再转成Blob后转换为ArrayBuffer发送。
3. 定时发送连续帧：视频是连续画面，需要循环捕获并发送帧，保持流的连续性。

修改后的React代码：

import { useRef, useEffect } from 'react';

function VideoStreamSender() {
  const videoRef = useRef(null);
  const canvasRef = useRef(null);
  const socketRef = useRef(null);

  useEffect(() => {
    navigator.mediaDevices.getUserMedia({ audio: false, video: true })
      .then(stream => {
        videoRef.current.srcObject = stream;
        
        // 初始化WebSocket连接
        socketRef.current = new WebSocket('ws://localhost:8000/ws');
        socketRef.current.onopen = () => {
          console.log('WebSocket连接成功');
          // 启动帧发送循环
          sendVideoFrames();
        };

        socketRef.current.onmessage = (event) => {
          console.log("[message] 收到服务器响应:", event.data);
        };

        socketRef.current.onclose = () => {
          console.log("连接已关闭");
        };

        socketRef.current.onerror = () => {
          console.log("WebSocket错误");
        };
      });

    const sendVideoFrames = () => {
      // 检查必要元素是否就绪
      if (!videoRef.current || !canvasRef.current || !socketRef.current || socketRef.current.readyState !== WebSocket.OPEN) {
        return;
      }

      const canvas = canvasRef.current;
      const ctx = canvas.getContext('2d');
      // 同步Canvas尺寸和视频实际尺寸
      canvas.width = videoRef.current.videoWidth;
      canvas.height = videoRef.current.videoHeight;
      // 将当前视频帧绘制到Canvas
      ctx.drawImage(videoRef.current, 0, 0, canvas.width, canvas.height);
      
      // 把Canvas内容转成JPEG Blob，再转成ArrayBuffer发送
      canvas.toBlob(async (blob) => {
        const arrayBuffer = await blob.arrayBuffer();
        socketRef.current.send(arrayBuffer);
      }, 'image/jpeg', 0.8);

      // 循环发送下一帧（约30fps）
      requestAnimationFrame(sendVideoFrames);
    };

    return () => {
      if (socketRef.current) {
        socketRef.current.close();
      }
    };
  }, []);

  return (
    <div>
      <video controls style={{ height: 500, width: 500 }} autoPlay ref={videoRef} />
      {/* 隐藏Canvas，仅用于捕获帧 */}
      <canvas ref={canvasRef} style={{ display: 'none' }} />
    </div>
  );
}

export default VideoStreamSender;

后端（Python）修改要点 #

1. 移除多余参数：原代码的websocket_endpoint函数多了data_image参数，FastAPI的WebSocket端点不需要额外的无定义参数，直接移除即可。
2. 优化异常处理：不要直接忽略异常，打印错误信息方便调试。
3. 完善图像校验：增加图像解码失败的判断，避免后续逻辑出错。

修改后的Python代码：

from fastapi import FastAPI, WebSocket
import numpy as np
import cv2

app = FastAPI()

@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    print("正在接受连接")
    await websocket.accept()
    print('连接已接受')
    try:
        while True:
            # 接收前端发送的ArrayBuffer（JPEG字节数据）
            image_bytes = await websocket.receive_bytes()
            print('收到图像数据')
            
            # 转换为OpenCV可处理的格式
            nparr = np.frombuffer(image_bytes, np.uint8)
            img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
            
            if img is None:
                print('图像解码失败')
                continue
            
            # 这里添加你的业务逻辑，比如图像分析、目标检测等
            # 示例：给图像添加文字水印
            cv2.putText(img, "已处理", (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2)
            
            # 将处理后的图像编码回JPEG，发送回前端
            _, buffer = cv2.imencode('.jpg', img)
            response_bytes = buffer.tobytes()
            await websocket.send_bytes(response_bytes)
    except Exception as e:
        print(f"连接异常: {e}")
    finally:
        print('连接已关闭')

注意事项 #

• 前端要确保视频加载完成后再开始捕获帧，否则videoWidth和videoHeight可能为0导致Canvas绘制失败。
• 可以根据需求调整发送帧率，比如用setInterval替代requestAnimationFrame来控制发送频率，避免带宽占用过高。
• 后端处理高帧率视频时，建议使用异步任务或线程池优化性能，避免阻塞事件循环。

内容的提问来源于stack exchange，提问作者Indradeep