Windows下Python接收自定义编码RTSP流的问题求助

问题背景 #

我在Windows上开发PySide应用，需要接收携带自定义编码数据的RTSP流，但多次尝试均失败。每个RTP数据包共21字节，最后9字节存储两个浮点数和一个布尔值，我只需获取数据包自行处理，无需自定义GStreamer插件。偏好GStreamer是因其延迟低、稳定性高，设备自带API性能极差。

尝试1：OpenCV + GStreamer后端 #

此前用该方式接收普通视频流正常，但自定义编码流无法打开。测试代码：

import os
import cv2

gst = f"rtspsrc location={rtsp_url} ! appsink"

os.environ["GST_DEBUG"] = "3"
cap = cv2.VideoCapture(gst, cv2.CAP_GSTREAMER)

if cap.isOpened():
    while True:
        ret, frame = cap.read()

        key = cv2.waitKey(1)

        if key & 0xFF == ord('q'):
            break

        print(frame)
else:
    print("failed to open stream")

运行输出：

0:00:00.156754700 25048 0000020BB33DCBC0 FIXME                default gstutils.c:4089:gst_element_decorate_stream_id_internal:<fakesrc0> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
0:00:03.694289100 25048 0000020BB38135C0 WARN                 default gst/parse/grammar.y:918:gst_parse_no_more_pads:<rtspsrc0> warning: Delayed linking failed.
0:00:03.698133300 25048 0000020BB38135C0 WARN                 default gst/parse/grammar.y:918:gst_parse_no_more_pads:<rtspsrc0> warning: failed delayed linking some pad of GstRTSPSrc named rtspsrc0 to some pad of GstAppSink named appsink0
0:00:03.705085500 25048 0000020BB33DCC40 WARN                 basesrc gstbasesrc.c:3177:gst_base_src_loop:<udpsrc1> error: Internal data stream error.
0:00:03.708917300 25048 0000020BB33DCC40 WARN                 basesrc gstbasesrc.c:3177:gst_base_src_loop:<udpsrc1> error: streaming stopped, reason not-linked (-1)
[ WARN:0@4.657] global cap_gstreamer.cpp:2839 cv::handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module udpsrc1 reported: Internal data stream error.
[ WARN:0@4.666] global cap_gstreamer.cpp:1698 cv::GStreamerCapture::open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0@4.667] global cap_gstreamer.cpp:1173 cv::GStreamerCapture::isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
[ WARN:0@4.667] global cap.cpp:206 cv::VideoCapture::open VIDEOIO(GSTREAMER): backend is generally available but can't be used to capture by name
failed to open stream

该GStreamer管道在命令行用gst-launch-1.0运行正常，我猜测cv2.VideoCapture仅支持标准视频帧格式，但不确定。

尝试2：GStreamer Python绑定 #

未找到适用于Windows的版本，WSL中测试代码可正常运行，但开发进度已深入，不愿迁移环境。测试代码：

import gi
from threading import Thread
import struct

gi.require_version('Gst', '1.0')
gi.require_version("GstApp", "1.0")

from gi.repository import Gst, GstApp, GLib

_ = GstApp

Gst.init()

main_loop = GLib.MainLoop()
main_loop_thread = Thread(target=main_loop.run)
main_loop_thread.start()

pipeline = Gst.parse_launch("rtspsrc location=rtsp_url ! appsink name=sink")

appsink = pipeline.get_by_name("sink")

pipeline.set_state(Gst.State.PLAYING)

try:
    while True:
        sample = appsink.try_pull_sample(Gst.SECOND)
        if sample is None:
            print("no sample")
            continue
        
        buf = sample.get_buffer()
        result, mapinfo = buf.map(Gst.MapFlags.READ)
        dat = mapinfo.data

        datum = dat[12:]

        x, y, worn = struct.unpack("!ffB", datum)
        print(x, y)

except KeyboardInterrupt:
    print("stopping")
    pass


pipeline.set_state(Gst.State.NULL)
main_loop.quit()
main_loop_thread.join()

需要解决的问题：

1. 能否让OpenCV适配该场景？
2. 有无方法在Windows上运行GStreamer的Python代码？

解决方案 #

一、OpenCV适配的可行性 #

你的猜测正确：cv2.VideoCapture基于GStreamer后端时，仅支持标准视频媒体类型（如H.264、MJPEG等），它内部会自动尝试连接视频解码元素，而自定义编码流不属于标准视频格式，导致管道链接失败。这种场景下OpenCV无法适配，因为它的GStreamer集成是为视频处理设计的，没有透传自定义RTP数据包的机制。

二、在Windows上运行GStreamer Python绑定的步骤 #

1. 安装GStreamer环境

• 下载Windows版本的GStreamer安装包，选择与Python架构一致的版本（如MSVC 64-bit），同时安装Runtime Installer和Development Installer。
• 将GStreamer的bin目录（如C:\gstreamer\1.0\msvc_x86_64\bin）添加到系统环境变量PATH中。

2. 安装PyGObject绑定

打开命令提示符，执行：

pip install pycairo pygobject

注意：必须保证Python与GStreamer架构一致（均为64位或32位），否则会出现兼容性问题。

3. 验证环境

运行以下代码测试：

import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst
Gst.init(None)
print(Gst.version_string())

若能正常输出版本信息，说明环境搭建成功。

4. 调整原有代码

给appsink添加明确的caps参数，指定RTP负载类型（根据你的流实际配置修改），同时关闭同步降低延迟：

import gi
from threading import Thread
import struct

gi.require_version('Gst', '1.0')
gi.require_version("GstApp", "1.0")

from gi.repository import Gst, GstApp, GLib

_ = GstApp

Gst.init()

main_loop = GLib.MainLoop()
main_loop_thread = Thread(target=main_loop.run)
main_loop_thread.start()

# 替换为你的RTSP地址，payload值根据实际流配置修改
pipeline = Gst.parse_launch("rtspsrc location=你的RTSP地址 ! appsink name=sink caps=\"application/x-rtp,payload=96\" sync=false")

appsink = pipeline.get_by_name("sink")

pipeline.set_state(Gst.State.PLAYING)

try:
    while True:
        sample = appsink.try_pull_sample(Gst.SECOND)
        if sample is None:
            print("no sample")
            continue
        
        buf = sample.get_buffer()
        result, mapinfo = buf.map(Gst.MapFlags.READ)
        dat = mapinfo.data

        datum = dat[12:]
        x, y, worn = struct.unpack("!ffB", datum)
        print(x, y)

except KeyboardInterrupt:
    print("stopping")
    pass

pipeline.set_state(Gst.State.NULL)
main_loop.quit()
main_loop_thread.join()

三、替代方案：Python子进程调用GStreamer命令行 #

如果上述方法遇到问题，可直接通过Python启动gst-launch-1.0，捕获输出并处理：

import subprocess
import struct

# 替换为你的RTSP地址，payload值根据实际流配置修改
cmd = [
    "gst-launch-1.0",
    "rtspsrc", f"location=你的RTSP地址",
    "!", "appsink", "emit-signals=true", "sync=false", "caps=\"application/x-rtp,payload=96\"", "dump=true"
]

proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)

try:
    while True:
        data = proc.stdout.read(21)
        if not data:
            break
        datum = data[12:]
        x, y, worn = struct.unpack("!ffB", datum)
        print(x, y)
except KeyboardInterrupt:
    proc.terminate()
    proc.wait()

该方法无需安装GStreamer Python绑定，直接利用命令行工具快速实现需求。

内容的提问来源于stack exchange，提问作者ismail hosny