使用GStreamer Compositor合并shmsrc视频时GST_FORMAT_TIME断言失败求助

问题 #

我有多个通过shmsink发送原始视频的GStreamer管道，命令如下：

gst-launch-1.0 videotestsrc ! video/x-raw,format=I420,width=640,height=360,framerate=15/1 ! timeoverlay ! queue ! shmsink socket-path=/tmp/test1 shm-size=20000000 sync=true

gst-launch-1.0 videotestsrc ! video/x-raw,format=I420,width=640,height=360,framerate=15/1 ! timeoverlay ! queue ! shmsink socket-path=/tmp/test2 shm-size=20000000 sync=true

尝试在另一个进程中用compositor插件合并视频，管道命令如下：

gst-launch-1.0 shmsrc socket-path=/tmp/test1 is-live=true ! queue ! video/x-raw,format=I420,width=640,height=360 ! compositor name=comp sink_1::xpos=860 ! videoconvert ! autovideosink \
shmsrc socket-path=/tmp/test2 is-live=true ! queue ! video/x-raw,format=I420,width=640,height=360 ! comp.

但触发GST_FORMAT_TIME不可用的断言错误：

ERROR:../subprojects/gst-plugins-base/gst-libs/gst/video/gstvideoaggregator.c:2322:gst_video_aggregator_sink_event: assertion failed: (seg.format == GST_FORMAT_TIME)
Bail out! ERROR:../subprojects/gst-plugins-base/gst-libs/gst/video/gstvideoaggregator.c:2322:gst_video_aggregator_sink_event: assertion failed: (seg.format == GST_FORMAT_TIME)
[1]    268025 abort (core dumped)  GST_DEBUG=4 gst-launch-1.0 shmsrc socket-path=/tmp/test1 is-live=true !

该问题在通过gstreamer-rs（GStreamer Rust绑定）编程实现时也会复现。试过videomixer元素同样报错，插入identity sync=true也无法解决，请问是否有手动添加GST_FORMAT_TIME的方法？

解决方案 #

这个错误的核心是shmsrc默认不会给接收的缓冲区打时间戳，而compositor/videomixer这类基于GstVideoAggregator的元素要求输入流必须携带GST_FORMAT_TIME格式的分段事件。以下是两种可行的解决方法：

1. 命令行管道修改 #

给接收端的每个shmsrc添加do-timestamp=true参数，让它为缓冲区生成时间戳，这样就能触发正确的GST_FORMAT_TIME分段事件。修改后的接收管道命令：

gst-launch-1.0 shmsrc socket-path=/tmp/test1 is-live=true do-timestamp=true ! queue ! video/x-raw,format=I420,width=640,height=360,framerate=15/1 ! compositor name=comp sink_0::xpos=0 sink_1::xpos=640 ! videoconvert ! autovideosink \
shmsrc socket-path=/tmp/test2 is-live=true do-timestamp=true ! queue ! video/x-raw,format=I420,width=640,height=360,framerate=15/1 ! comp.

2. gstreamer-rs编程实现调整 #

在代码中为每个ShmSrc元素设置do-timestamp属性为true，同时确保caps包含正确的帧率信息，示例代码片段：

use gst::prelude::*;

fn main() {
    gst::init().unwrap();

    let pipeline = gst::Pipeline::new(None);

    // 初始化第一个shmsrc流
    let shmsrc1 = gst::ElementFactory::make("shmsrc", None).unwrap();
    shmsrc1.set_property("socket-path", "/tmp/test1").unwrap();
    shmsrc1.set_property("is-live", true).unwrap();
    shmsrc1.set_property("do-timestamp", true).unwrap();

    let queue1 = gst::ElementFactory::make("queue", None).unwrap();
    let capsfilter1 = gst::ElementFactory::make("capsfilter", None).unwrap();
    let caps1 = gst::Caps::builder("video/x-raw")
        .field("format", "I420")
        .field("width", 640)
        .field("height", 360)
        .field("framerate", gst::Fraction::new(15, 1))
        .build();
    capsfilter1.set_property("caps", caps1).unwrap();

    // 初始化第二个shmsrc流
    let shmsrc2 = gst::ElementFactory::make("shmsrc", None).unwrap();
    shmsrc2.set_property("socket-path", "/tmp/test2").unwrap();
    shmsrc2.set_property("is-live", true).unwrap();
    shmsrc2.set_property("do-timestamp", true).unwrap();

    let queue2 = gst::ElementFactory::make("queue", None).unwrap();
    let capsfilter2 = gst::ElementFactory::make("capsfilter", None).unwrap();
    let caps2 = gst::Caps::builder("video/x-raw")
        .field("format", "I420")
        .field("width", 640)
        .field("height", 360)
        .field("framerate", gst::Fraction::new(15, 1))
        .build();
    capsfilter2.set_property("caps", caps2).unwrap();

    // 初始化compositor和输出组件
    let compositor = gst::ElementFactory::make("compositor", None).unwrap();
    compositor.set_property_from_str("sink_0::xpos", "0").unwrap();
    compositor.set_property_from_str("sink_1::xpos", "640").unwrap();

    let videoconvert = gst::ElementFactory::make("videoconvert", None).unwrap();
    let autovideosink = gst::ElementFactory::make("autovideosink", None).unwrap();

    // 将元素添加到管道并链接
    pipeline.add_many(&[&shmsrc1, &queue1, &capsfilter1, &shmsrc2, &queue2, &capsfilter2, &compositor, &videoconvert, &autovideosink]).unwrap();
    gst::Element::link_many(&[&shmsrc1, &queue1, &capsfilter1, &compositor]).unwrap();
    gst::Element::link_many(&[&shmsrc2, &queue2, &capsfilter2, &compositor]).unwrap();
    gst::Element::link_many(&[&compositor, &videoconvert, &autovideosink]).unwrap();

    // 启动管道并等待结束
    pipeline.set_state(gst::State::Playing).unwrap();

    let bus = pipeline.bus().unwrap();
    for msg in bus.iter_timed(gst::ClockTime::NONE) {
        match msg.view() {
            gst::MessageView::Error(err) => {
                eprintln!("Error from {:?}: {} ({:?})", err.src().map(|s| s.path_string()), err.error(), err.debug());
                break;
            }
            gst::MessageView::Eos(_) => break,
            _ => (),
        }
    }

    pipeline.set_state(gst::State::Null).unwrap();
}

通过上述设置，shmsrc会为每个缓冲区生成时间戳，compositor就能收到符合要求的GST_FORMAT_TIME分段事件，避免断言错误。

内容的提问来源于stack exchange，提问作者Gatothgaj