GStreamer中appsrc的原始BGR流在autovideosink中失真的原因排查

我开发了一个C++ GStreamer应用，通过appsrc元素将原始8位BGR帧加载到管道中——注意AirSim的MultirotorRpcLibClient.simGetImages()函数文档标注返回RGB，但实际实现是BGR。

simGetImages()返回的结构体包含一个uint8向量属性，帧默认尺寸为256×144像素，每个像素3个通道，单帧总大小110592字节。我已经验证过AirSim生成的原始BGR数据：单帧大小正确，通道值有效；相机静止时像素值稳定，说明原始数据没有偏移问题。

我的GStreamer管道设计为：appsrc -> queue -> rawvideoparse -> videoconvert -> autovideosink。原始BGR字节被加载到缓冲区后推送到appsrc，每个缓冲区包含一整帧数据。管道能将帧流传输到autovideosink，但画面经常出现左移或失真（如图所示）。

请问是数据加载到缓冲区并发送到appsrc的方式有问题？还是需要指定额外的Caps让GStreamer正确渲染帧？

管道初始化代码 #

typedef struct _PipelineData {
  GstElement *pipeline, *app_source, *app_sink, *queue_0;

  GMainLoop *main_loop;  /* GLib's Main Loop */
} PipelineData;


static int runGstreamer(int *argc, char **argv[], PipelineData *data) {
    GstBus *bus;
    guint bus_watch_id;

    // initialize gstreamer
    gst_init(argc, argv);

    // Create the elements
    data->app_source = gst_element_factory_make ("appsrc", "video_source");
    data->queue_0 = gst_element_factory_make ("queue", "queue_0");
    data->app_sink = gst_element_factory_make ("autovideosink", "video_sink");

    // create empty pipeline
    data->pipeline = gst_pipeline_new ("video-pipeline");

 
    // element configuration goes here 
    g_object_set(G_OBJECT(data->app_source),
                "format", 3,
                "is-live", true,
                NULL);

    // link elements
    gst_bin_add_many(
        GST_BIN (data->pipeline),
        data->app_source,
        data->queue_0,
        data->app_sink,
        NULL);
    
    GstCaps *caps_source;

    caps_source = gst_caps_new_simple ("video/x-raw",
            "format", G_TYPE_STRING, "BGR",
            "framerate", GST_TYPE_FRACTION, 1, 1,
            "width", G_TYPE_INT, 256,
            "height", G_TYPE_INT, 144,
            NULL);

    if (!gst_element_link_filtered(data->app_source, data->queue_0, caps_source)) {
        g_printerr("Elements app_source and queue_0 could not be linked.\n");
        gst_object_unref (data->pipeline);
        return -1;
    }
    gst_caps_unref(caps_source);

    
    if (gst_element_link_many (data->queue_0, data->app_sink, NULL) != TRUE) {
        g_printerr("Elements could not be linked.\n");
        gst_object_unref (data->pipeline);
        return -1;
    }

    // start playing the pipeline
    gst_element_set_state (data->pipeline, GST_STATE_PLAYING);

    // create and start main loop
    // add a message handler
    data->main_loop = g_main_loop_new (NULL, FALSE);

    bus = gst_pipeline_get_bus (GST_PIPELINE (data->pipeline));
    bus_watch_id = gst_bus_add_watch (bus, bus_call, data->main_loop);
    gst_object_unref (bus);

    g_main_loop_run (data->main_loop);

    // Free resources
    gst_element_set_state (data->pipeline, GST_STATE_NULL);
    gst_object_unref (data->pipeline);
    return 0;
}

图像数据加载函数 #

static void sendImageStream(PipelineData * pipelineData, int fps) {
    printf("Milliseconds between frames: %d\n", (int)((1 / (float) fps) * 1e3));

    unsigned long frame_count = 1;
    while(1) {
        vector<uint8_t> newImage = getOneImage(frame_count);
        
        // check that appsrc element is created in gstreamer thread before using
        if (pipelineData->app_source) {
            GstBuffer *buffer;
            GstMapInfo map;
            GstFlowReturn ret;
            
            // create buffer and allocate memory
            buffer = gst_buffer_new_allocate(NULL, (gint)newImage.size(), NULL);
            // set image presentation timestamp in nanoseconds
            GST_BUFFER_TIMESTAMP(buffer) = frame_count * 1e9 / fps;
            // fill writable map with (ideally writable) memory blocks in the buffer
            gst_buffer_map(buffer, &map, GST_MAP_WRITE);
            map.data = newImage.data();
            map.size = newImage.size();
            ret = gst_app_src_push_buffer(GST_APP_SRC(pipelineData->app_source), buffer);
            // release buffer memory that was associated with map
            gst_buffer_unmap(buffer, &map);
            // see flow error type of GstFlowReturn
            if (ret != 0) {
                g_print("\nPush appsrc buffer flow error: %d\n", ret);
            }
        }
        else {
            std::cout << "AppSrc element not yet created - image skipped" << std::endl;
        }
        // std::cout << "\nImage unit8 size: " << newImage.size() << std::endl;
        std::this_thread::sleep_for(std::chrono::milliseconds((int)((1 / (float) fps) * 1e3)));

        frame_count++;
    }
}

主函数 #

int main(int argc, char *argv[]) {
    PipelineData data = {};
    
    std::thread feedAppSrc(sendImageStream, &data, 1);

    int pipelineStatus = runGstreamer(&argc, &argv, &data);

    if (!pipelineStatus) {
        feedAppSrc.join();
    }

    if (pipelineStatus) {
        std::cout << "\nPipeline failed to run: terminating feedAppSrc and the program" << std::endl;
    }

    return pipelineStatus;
}

1. 缓冲区数据拷贝错误 #

在sendImageStream函数中，直接修改GstMapInfo的data指针并没有将AirSim的图像数据拷贝到GStreamer缓冲区，而是让缓冲区指向了局部变量newImage的内存。newImage在函数执行结束后会被释放，导致GStreamer访问无效内存，引发画面失真。

修复：使用memcpy将数据拷贝到缓冲区的映射内存：

// 替换原map指针修改逻辑
gst_buffer_map(buffer, &map, GST_MAP_WRITE);
memcpy(map.data, newImage.data(), newImage.size());
gst_buffer_unmap(buffer, &map);

2. 缺失管道关键元素 #

你设计的管道包含rawvideoparse和videoconvert，但代码中完全没有创建这两个元素！GStreamer无法正确解析原始帧的边界和格式，必然出现画面偏移、撕裂问题。

修复：添加缺失元素并正确链接：

1. 更新PipelineData结构体：

typedef struct _PipelineData {
  GstElement *pipeline, *app_source, *app_sink, *queue_0, *rawvideoparse, *videoconvert;
  GMainLoop *main_loop;
} PipelineData;

2. 创建并链接元素：

// 在runGstreamer函数的元素创建部分添加
data->rawvideoparse = gst_element_factory_make("rawvideoparse", "raw_parse");
data->videoconvert = gst_element_factory_make("videoconvert", "conv");

// 添加到管道时包含新元素
gst_bin_add_many(
    GST_BIN (data->pipeline),
    data->app_source, data->queue_0, data->rawvideoparse, data->videoconvert, data->app_sink,
    NULL);

// 修改链接逻辑，按设计的管道顺序链接
if (!gst_element_link_filtered(data->app_source, data->queue_0, caps_source)) {
    // 原有错误处理
}
if (!gst_element_link_many(data->queue_0, data->rawvideoparse, data->videoconvert, data->app_sink, NULL)) {
    g_printerr("Elements could not be linked.\n");
    gst_object_unref (data->pipeline);
    return -1;
}

3. AppSrc格式参数不规范 #

直接使用数字3设置format参数可读性差，应该使用GStreamer枚举值GST_FORMAT_TIME：

g_object_set(G_OBJECT(data->app_source),
            "format", GST_FORMAT_TIME,
            "is-live", TRUE,
            NULL);

4. 线程竞态优化 #

子线程可能在app_source初始化完成前就尝试推送数据，虽然有空指针检查，但可以通过Glib互斥锁进一步保证线程安全：

1. 在PipelineData中添加互斥锁成员：GMutex mutex;
2. 在runGstreamer初始化时加锁：g_mutex_init(&data->mutex);
3. 访问app_source时加锁：

g_mutex_lock(&pipelineData->mutex);
if (pipelineData->app_source) {
    // 推送缓冲区逻辑
}
g_mutex_unlock(&pipelineData->mutex);

内容的提问来源于stack exchange，提问作者Matt Brauer