如何在RTSP服务器的Gstreamer流水线中结合socketsrc用OpenCV处理帧

核心架构：拆分流水线+双向数据交互 #

将原有RTSP流水线拆分为接收处理段和推送输出段，通过appsink提取帧给OpenCV处理，再用appsrc将处理后的帧回灌到推送段，保证数据流的连续与格式一致性。

1. 接收段：从Socketsrc到Appsink的帧提取 #

• 配置appsink开启信号触发，监听new-sample回调来获取解码后的原始帧：

  static GstFlowReturn on_new_sample(GstAppSink* sink, gpointer user_data) {
      GstSample* sample = gst_app_sink_pull_sample(sink);
      GstBuffer* buffer = gst_sample_get_buffer(sample);
      GstMapInfo map;
      
      // 映射缓冲区数据到OpenCV Mat
      gst_buffer_map(buffer, &map, GST_MAP_READ);
      cv::Mat frame(720, 1280, CV_8UC3, map.data, map.stride); // 根据实际分辨率调整
      
      // 调用OpenCV处理逻辑
      cv::cvtColor(frame, frame, cv::COLOR_BGR2GRAY); // 示例：灰度化处理
      
      // 释放资源
      gst_buffer_unmap(buffer, &map);
      gst_sample_unref(sample);
      
      // 将处理后的帧推送到appsrc（需全局或通过user_data传递appsrc实例）
      push_processed_frame(frame);
      return GST_FLOW_OK;
  }
  

• 提前固定appsink的Caps参数，避免格式协商混乱：

  # 流水线示例
  socketsrc port=5000 ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! video/x-raw,format=BGR,width=1280,height=720,framerate=30/1 ! appsink name=frame_sink emit-signals=true
  

2. 推送段：Appsrc到RTSP输出的帧回灌 #

• 初始化appsrc时设置直播模式与格式匹配：

  GstElement* appsrc = gst_element_factory_make("appsrc", "frame_src");
  g_object_set(G_OBJECT(appsrc), "is-live", TRUE, "format", GST_FORMAT_TIME, NULL);
  
  // 设置与appsink一致的Caps
  GstCaps* caps = gst_caps_from_string("video/x-raw,format=BGR,width=1280,height=720,framerate=30/1");
  g_object_set(G_OBJECT(appsrc), "caps", caps, NULL);
  gst_caps_unref(caps);
  

• 实现帧推送函数，将OpenCV处理后的Mat封装为GstBuffer：

  void push_processed_frame(cv::Mat& frame) {
      GstBuffer* buffer = gst_buffer_new_allocate(NULL, frame.total() * frame.elemSize(), NULL);
      GstMapInfo map;
      
      gst_buffer_map(buffer, &map, GST_MAP_WRITE);
      memcpy(map.data, frame.data, frame.total() * frame.elemSize());
      gst_buffer_unmap(buffer, &map);
      
      // 设置时间戳保证流时序稳定
      GST_BUFFER_PTS(buffer) = gst_util_get_timestamp();
      GST_BUFFER_DURATION(buffer) = gst_util_uint64_scale_int(1, GST_SECOND, 30); // 30fps
      
      GstFlowReturn ret;
      g_signal_emit_by_name(appsrc, "push-buffer", buffer, &ret);
      gst_buffer_unref(buffer);
  }
  

• 推送段流水线衔接RTSP输出：

  appsrc name=frame_src ! videoconvert ! x264enc tune=zerolatency ! rtph264pay ! udpsink host=0.0.0.0 port=5001
  

3. RTSP服务器整合与同步 #

• 使用gst-rtsp-server模块创建服务，将推送段流水线注册为媒体流：

  GstRTSPServer* server = gst_rtsp_server_new();
  GstRTSPMountPoints* mounts = gst_rtsp_server_get_mount_points(server);
  GstRTSPMediaFactory* factory = gst_rtsp_media_factory_new();
  
  // 设置推送段流水线描述
  gst_rtsp_media_factory_set_launch(factory, "( appsrc name=frame_src ! videoconvert ! x264enc tune=zerolatency ! rtph264pay name=pay0 pt=96 )");
  
  gst_rtsp_mount_points_add_factory(mounts, "/stream", factory);
  g_object_unref(mounts);
  gst_rtsp_server_attach(server, NULL);
  

• 统一两段流水线的时钟，避免时序偏移：

  GstClock* system_clock = gst_system_clock_obtain();
  gst_pipeline_use_clock(GST_PIPELINE(receive_pipeline), system_clock);
  gst_pipeline_use_clock(GST_PIPELINE(push_pipeline), system_clock);
  gst_object_unref(system_clock);
  

4. 关键优化点 #

• 格式转换：用videoconvert节点在解码后/编码前做格式适配，避免OpenCV与GStreamer格式冲突。
• 性能优化：将OpenCV处理逻辑放到独立线程，避免阻塞GStreamer的消息循环；对高分辨率帧可采用多线程并行处理。
• 内存管理：严格调用gst_sample_unref、gst_buffer_unref释放资源，防止内存泄漏。

内容的提问来源于stack exchange，提问作者Roy Amoyal