如何实现Python与Rust通过UNIX Socket传输视频流？

需求 #

构建一套通过UNIX IPC Socket实现从Python到Rust的视频流传输系统，Python独占摄像头/视频资源，开发环境为Rust新手。

已尝试流程 #

• 在Python中提取视频每一帧并转换为NumPy数组
• 将数组转为字符串再转为字节
• 通过UNIX IPC Socket发送
• 在接收端（Rust）转换回字符串
• 尝试解析为可用数组并生成图像

发送端（Python）代码 #

import socket, os, cv2, numpy
# print all the string without truncating
numpy.set_printoptions(threshold=numpy.inf)

# declare the camera socket and unlink path if already in use
camera_socket_path = '/home/user/exps/test.sock'
try:
    os.unlink(camera_socket_path)
except OSError:
    pass

# create and listen on the socket
camera_socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
camera_socket.bind(camera_socket_path)
camera_socket.listen()

conn, addr = camera_socket.accept()

vidcap = cv2.VideoCapture('example.mp4')
success, image = vidcap.read()
print(str(image))
while success:
    conn.send(bytes(str(image), 'utf-8'))      
    success, image = vidcap.read()
    break # after sending a frame, for testing

conn.close()

接收端（Rust）代码 #

use std::os::unix::net::UnixStream;
use std::io::prelude::*;

use image::RgbImage;
use ndarray::Array3;

fn array_to_image(arr: Array3<u8>) -> RgbImage {
    assert!(arr.is_standard_layout());

    let (height, width, _) = arr.dim();
    let raw = arr.into_raw_vec();

    RgbImage::from_raw(width as u32, height as u32, raw)
        .expect("container should have the right size for the image dimensions")
}

fn main() {
    // create a standard UNIX IPX socket stream
    let mut stream = UnixStream::connect("/home/user/exps/test.sock").unwrap();

    loop {
        // 2074139 is the length of the string printed at sender :D
        // I thought it might work out but it didn't obviously
        // the docs suggest an exponential of 2. default was 1024
        let mut buf = [0; 2074139];
        let count = stream.read(&mut buf).unwrap();
        let response = String::from_utf8(buf[..2074139].to_vec()).unwrap();
        // write a parser function to convert the string to Array3 iterating through line()
        // let pic = array_to_image(convert(response));
        println!("{}", response);
        break; // get a single frame, for testing
    }
}

当前问题 #

收发的字符串完全不一致，无法正确解析图像数据，最终目标是实现基于摄像头（如cv2.VideoCapture）的持续视频流传输。

需要完成的工作 #

1. 替换低效的字符串序列化方案 #

将NumPy数组转字符串的方式替换为二进制序列化，直接传输原始像素数据，避免字符串转换带来的格式错乱和性能损耗：

• Python端修改：用numpy.ndarray.tobytes()直接生成原始字节，同时先发送帧的元数据（宽、高、通道数），让Rust端知道如何重构数组。示例修改：

  import struct
  
  # ... 原有Socket初始化代码 ...
  
  vidcap = cv2.VideoCapture('example.mp4')
  success, image = vidcap.read()
  while success:
      # 打包帧元数据：宽、高、通道数（大端序4字节整数）
      height, width, channels = image.shape
      meta = struct.pack('>III', width, height, channels)
      # 确保元数据完整发送
      conn.sendall(meta)
      # 发送帧原始字节数据
      conn.sendall(image.tobytes())
      
      success, image = vidcap.read()
      # 测试时保留break，正式环境移除
      break
  
  conn.close()
  

2. 修复Rust端的读取逻辑 #

UNIX Stream是字节流，需先读取元数据再按需读取帧数据，避免固定缓冲区带来的截断或冗余：

• 添加byteorder依赖到Cargo.toml，用于解析结构化元数据：

  [dependencies]
  byteorder = "1.4"
  ndarray = "0.15"
  image = "0.24"
  

• 修改Rust读取逻辑：

  use std::os::unix::net::UnixStream;
  use std::io::{Read, Error};
  use byteorder::{BigEndian, ReadBytesExt};
  use ndarray::Array3;
  use image::RgbImage;
  
  fn array_to_image(arr: Array3<u8>) -> RgbImage {
      assert!(arr.is_standard_layout());
      let (height, width, _) = arr.dim();
      let raw = arr.into_raw_vec();
      RgbImage::from_raw(width as u32, height as u32, raw)
          .expect("container should have the right size for the image dimensions")
  }
  
  fn main() -> Result<(), Box<dyn std::error::Error>> {
      let mut stream = UnixStream::connect("/home/user/exps/test.sock")?;
  
      // 读取元数据：宽、高、通道数
      let width = stream.read_u32::<BigEndian>()?;
      let height = stream.read_u32::<BigEndian>()?;
      let channels = stream.read_u32::<BigEndian>()?;
  
      // 计算帧数据总字节数
      let data_len = (width * height * channels) as usize;
      let mut buf = vec![0u8; data_len];
      // 确保完整读取所有帧数据
      stream.read_exact(&mut buf)?;
  
      // 重构3D数组
      let arr = Array3::from_shape_vec(
          (height as usize, width as usize, channels as usize), 
          buf
      )?;
  
      // 转换为图像并保存
      let img = array_to_image(arr);
      img.save("received_frame.png")?;
  
      Ok(())
  }
  

3. 处理持续流的帧边界 #

对于持续视频流，需在每帧前添加明确的分隔标识，避免帧数据粘包：

• 方案1：每帧开头先发送帧的总长度（元数据长度+数据长度），Rust端先读取长度再读取对应字节数；
• 方案2：每帧都发送完整的元数据+数据，Rust端循环读取元数据→读取帧数据的流程。

4. 完善错误处理与资源管理 #

• Python端：用sendall()代替send()确保数据完整发送，添加异常捕获处理Socket断开、摄像头故障等情况；
• Rust端：移除unwrap()，改用Result和?处理错误，确保Socket、文件等资源自动释放。

5. 可选性能优化 #

• 共享内存：使用UNIX共享内存（Python的mmap、Rust的memmap2）代替Socket，减少数据拷贝；
• 帧压缩：Python端用cv2.imencode('.jpg', image)将帧转为JPEG字节，Rust端用image::load_from_memory解码，降低传输数据量。

内容的提问来源于stack exchange，提问作者Vishal DS