GStreamer基于时间的图像叠加功能失效技术求助

我用Python写了GStreamer管道，想给MP4视频添加图像叠加效果，但输出视频和输入完全一致。启用GST_DEBUG=3后得到以下日志：

0:00:00.046586337 11711 0x279d380 WARN gdkpixbufoverlay gstgdkpixbufoverlay.c:562:gst_gdk_pixbuf_overlay_start:<gdkpixbufoverlay0> no image location set, doing nothing
0:00:00.047215851 11711 0x2766360 FIXME videodecoder gstvideodecoder.c:1193:gst_video_decoder_drain_out:<pngdec0> Sub-class should implement drain()
0:00:00.047218585 11711 0x279d380 WARN basesrc gstbasesrc.c:3688:gst_base_src_start_complete:<filesrc0> pad not activated yet
0:00:00.055638677 11711 0x263df00 WARN qtdemux qtdemux_types.c:249:qtdemux_type_get: unknown QuickTime node type sgpd
0:00:00.055691634 11711 0x263df00 WARN qtdemux qtdemux_types.c:249:qtdemux_type_get: unknown QuickTime node type sbgp
0:00:00.055736798 11711 0x263df00 WARN qtdemux qtdemux.c:3121:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 1
0:00:00.055879661 11711 0x263df00 WARN qtdemux qtdemux.c:3121:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 2
0:00:00.057660594 11711 0x2766360 WARN videodecoder gstvideodecoder.c:2816:gst_video_decoder_chain:<pngdec0> Received buffer without a new-segment. Assuming timestamps start from 0.
0:00:00.058040800 11711 0x2766360 WARN video-info video-info.c:760:gst_video_info_to_caps: invalid matrix 0 for RGB format, using RGB
0:00:00.205414894 11711 0x2766400 WARN audio-resampler audio-resampler.c:274:convert_taps_gint16_c: can't find exact taps
0:00:01.263245091 11711 0x27661e0 FIXME basesink gstbasesink.c:3395:gst_base_sink_default_event:<filesink0> stream-start event without group-id. Consider implementing group-id handling in the upstream elements
0:00:01.264908606 11711 0x27661e0 FIXME aggregator gstaggregator.c:1410:gst_aggregator_aggregate_func:<mux> Subclass should call gst_aggregator_selected_samples() from its aggregate implementation.
DEBUG:root:Position: 2.7s / 53.0s
DEBUG:root:Position: 4.333333333s / 53.0s

我的管道代码和目录结构如下：

管道代码 #

def start_pipeline(video_file_path: str, output_file_path: str) -> None:
    Gst.init(None)

    # GStreamer pipeline for adding image overlay to a video
    pipeline_string = (
        f"filesrc location={video_file_path} ! decodebin name=dec "
        f"dec. ! queue ! videoconvert ! x264enc ! queue ! mp4mux name=mux ! filesink location={output_file_path} "
        f'multifilesrc location=images/image_%06d.png index=1 caps="image/png,framerate=(fraction)30/1" ! pngdec ! videoconvert ! gdkpixbufoverlay ! queue ! x264enc ! queue ! mux. '
        f"dec. ! queue ! audioconvert ! audioresample ! voaacenc ! queue ! mux. "
    )
    pipeline = Gst.parse_launch(pipeline_string)

    # Set up bus to receive messages
    bus = pipeline.get_bus()
    bus.add_signal_watch()
    bus.connect("message", on_bus_message, GLib.MainLoop.new(None, False))

    # Start the pipeline
    pipeline.set_state(Gst.State.PLAYING)

    # Run the main loop
    loop = GLib.MainLoop()
    # Add a timeout callback to check the progress every second
    GLib.timeout_add_seconds(1, on_timeout, pipeline, loop)

    loop.run()
    loop.quit()
    exit("Done")

目录结构 #

.
├── images
│ ├── image_000000.png
│ ├── image_000001.png
│ ├── image_000002.png
│ ├── image_000003.png
│ ├── image_000004.png
│ ├── image_000005.png
│ ├── image_000006.png
│ ├── image_000007.png
│ ├── image_000008.png
│ └── image_000009.png
├── input.mp4
├── requirements.txt
├── stream.py

问题原因分析 #

1. gdkpixbufoverlay用法错误：日志核心警告no image location set, doing nothing说明该元素未生效。它的作用是在已有视频流上叠加单张本地图片（需通过location参数指定路径），不能直接接收图像流作为输入。
2. 管道结构错误：当前管道将原视频流、图像流分别编码后送入mp4mux，导致输出MP4包含两个独立视频流，播放时默认仅加载第一个（原视频流），因此看不到叠加效果。

解决方案 #

改用videomixer元素将原视频流与图像流混合为单一视频流，再编码封装。同时修正multifilesrc的index参数（图片从000000开始，index需设为0），并确保图像流的分辨率、帧率与原视频匹配。

修改后的管道代码 #

def start_pipeline(video_file_path: str, output_file_path: str) -> None:
    Gst.init(None)

    # 修正后的GStreamer管道：使用videomixer混合视频与图像
    pipeline_string = (
        f"filesrc location={video_file_path} ! decodebin name=dec "
        # 原视频流：解码后转格式、调整分辨率，作为videomixer底层输入
        f"dec. ! queue ! videoconvert ! videoscale ! video/x-raw,width=1920,height=1080 ! mix.sink_0 "
        # 图像流：读取序列帧，解码转格式、调整分辨率后作为叠加层输入
        f'multifilesrc location=images/image_%06d.png index=0 caps="image/png,framerate=(fraction)30/1" ! pngdec ! videoconvert ! videoscale ! video/x-raw,width=320,height=240 ! mix.sink_1 '
        # 混合后的视频流编码，与音频流一起封装
        f"videomixer name=mix sink_1::zorder=1 ! queue ! x264enc ! queue ! mp4mux name=mux ! filesink location={output_file_path} "
        # 音频流路径保持不变
        f"dec. ! queue ! audioconvert ! audioresample ! voaacenc ! queue ! mux. "
    )
    pipeline = Gst.parse_launch(pipeline_string)

    # 以下代码保持不变
    bus = pipeline.get_bus()
    bus.add_signal_watch()
    bus.connect("message", on_bus_message, GLib.MainLoop.new(None, False))

    pipeline.set_state(Gst.State.PLAYING)

    loop = GLib.MainLoop()
    GLib.timeout_add_seconds(1, on_timeout, pipeline, loop)

    loop.run()
    loop.quit()
    exit("Done")

关键调整说明 #

• videomixer混合流：用videomixer替代gdkpixbufoverlay，原视频作为底层（sink_0），图像作为叠加层（sink_1），通过zorder=1确保图像显示在视频上方。
• 分辨率匹配：通过videoscale和video/x-raw caps指定分辨率，可根据实际需求修改主视频与叠加图像的尺寸。
• multifilesrc index修正：图片文件名从image_000000.png开始，设置index=0才能正确读取第一张图。
• 帧率同步：确保图像流帧率（这里设为30/1）与原视频一致，若原视频帧率不同，可在图像流路径添加videorate ! video/x-raw,framerate=(fraction)XX/1调整。

内容的提问来源于stack exchange，提问作者Sagar Yadav