Jetson Nano下用Python3 OpenCV加速USB摄像头的问题排查

问题背景 #

需要在Jetson Nano上通过Python的cv2库，借助GStreamer的硬件加速插件（nvv4l2decoder、nvjpegdec、memory:NVMM）提升USB摄像头的处理效率，但当前代码运行报错。

现有Python代码 #

import cv2

width = 1920
height = 1080

gs_pipeline = f"v4l2src device=/dev/video0 io-mode=2 " \
              f"! image/jpeg, width={width}, height={height}" \
              f"! nvv4l2decoder mjpeg=1 " \
              f"! nvvidconv " \
              f"! video/x-raw(memory:NVMM) format=BGR" \
              f"! videoconvert " \
              f"! video/x-raw, format=BGR " \
              f"! appsink"

v_cap = cv2.VideoCapture(gs_pipeline, cv2.CAP_GSTREAMER)
if not v_cap.isOpened():
    print("failed to open video capture")
    exit(-1)

while v_cap.isOpened():
    ret_val, frame = v_cap.read()
    if not ret_val:
        break

    cv2.imshow('', frame)

    input_key = cv2.waitKey(1)
    if input_key != -1:
        print(f"input key = {input_key}")

    if input_key == ord('q'):
        break

运行报错信息 #

[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (711) open OpenCV | GStreamer warning: Error opening bin: could not parse caps "video/x-raw(memory:NVMM) format=BGR"
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

已知可行的GStreamer命令行示例 #

gst-launch-1.0 v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=1920, height=1080, framerate=30/1, format=MJPG ! nvjpegdec ! 'video/x-raw(memory:NVMM),format=I420,width=1920,height=1080,framerate=30/1' ! nvegltransform ! nveglglessink

问题分析 #

报错核心是GStreamer无法解析video/x-raw(memory:NVMM) format=BGR这部分caps，原因有两点：

1. 语法错误：NVMM格式的caps属性需要用逗号分隔，正确写法应为video/x-raw(memory:NVMM),format=XXX
2. 格式不支持：NVMM（GPU显存）中的原始视频格式仅支持YUV类（如I420、NV12），不支持BGR格式，无法直接将NVMM内存的帧转为BGR输出。

另外，OpenCV的appsink只能接收系统内存中的帧，而硬件解码插件输出的是NVMM内存帧，必须通过nvvidconv完成内存格式转换后才能传递给appsink。

修正后的代码 #

方案1：使用nvv4l2decoder加速解码 #

import cv2

width = 1920
height = 1080

# 修正后的GStreamer pipeline
gs_pipeline = f"v4l2src device=/dev/video0 io-mode=2 " \
              f"! image/jpeg, width={width}, height={height}, framerate=30/1 " \
              f"! nvv4l2decoder mjpeg=1 " \
              f"! nvvidconv ! video/x-raw, format=BGRx " \
              f"! videoconvert ! video/x-raw, format=BGR " \
              f"! appsink drop=1"

v_cap = cv2.VideoCapture(gs_pipeline, cv2.CAP_GSTREAMER)
if not v_cap.isOpened():
    print("failed to open video capture")
    exit(-1)

while v_cap.isOpened():
    ret_val, frame = v_cap.read()
    if not ret_val:
        break

    cv2.imshow('Camera', frame)

    input_key = cv2.waitKey(1)
    if input_key == ord('q'):
        break

v_cap.release()
cv2.destroyAllWindows()

方案2：使用nvjpegdec加速解码 #

import cv2

width = 1920
height = 1080

gs_pipeline = f"v4l2src device=/dev/video0 io-mode=2 " \
              f"! image/jpeg, width={width}, height={height}, framerate=30/1 " \
              f"! nvjpegdec " \
              f"! nvvidconv ! video/x-raw, format=BGRx " \
              f"! videoconvert ! video/x-raw, format=BGR " \
              f"! appsink drop=1"

v_cap = cv2.VideoCapture(gs_pipeline, cv2.CAP_GSTREAMER)
if not v_cap.isOpened():
    print("failed to open video capture")
    exit(-1)

while v_cap.isOpened():
    ret_val, frame = v_cap.read()
    if not ret_val:
        break

    cv2.imshow('Camera', frame)

    input_key = cv2.waitKey(1)
    if input_key == ord('q'):
        break

v_cap.release()
cv2.destroyAllWindows()

关键修改点说明 #

1. 移除错误的NVMM BGR caps：不再尝试直接在NVMM内存中生成BGR帧，改为先通过nvvidconv将NVMM内存的YUV帧转为系统内存的BGRx格式（BGR+alpha通道，是nvvidconv支持的系统内存输出格式）
2. 添加格式转换：通过videoconvert将BGRx转为OpenCV需要的纯BGR格式
3. 优化appsink：添加drop=1参数，当帧堆积时丢弃旧帧，避免延迟问题
4. 完善资源释放：添加v_cap.release()和cv2.destroyAllWindows()，避免内存泄漏

内容的提问来源于stack exchange，提问作者Jet C.