如何在视频流服务器中仅对I帧执行边缘检测（FFmpeg+OpenCV C++）

Hey there! Since you're already able to detect I-frames using FFmpeg, let's break down how to pair that with OpenCV C++ to generate edge maps—keep it simple, just like you asked.

Step 1: Convert FFmpeg's AVFrame to OpenCV Mat #

First, FFmpeg's decoded frames are usually in YUV format (most commonly YUV420P), but OpenCV works best with BGR. We'll use FFmpeg's sws_scale to convert the format, then load the data into an OpenCV Mat.

Pro tip: Initialize the scaling context once (not per frame) to save resources.

// Initialize the scaling context (do this once at the start of your program)
SwsContext* sws_ctx = sws_getContext(
    frame->width, frame->height, frame->format,  // Source frame specs
    frame->width, frame->height, AV_PIX_FMT_BGR24,  // Target: BGR for OpenCV
    SWS_BILINEAR, NULL, NULL, NULL  // Scaling algorithm + default options
);

// Create an OpenCV Mat to hold the converted BGR frame
cv::Mat bgr_frame(frame->height, frame->width, CV_8UC3);

// Prepare pointers/linesizes for the destination frame
uint8_t* dst_data[1] = { bgr_frame.data };
int dst_linesize[1] = { static_cast<int>(bgr_frame.step) };

// Convert the AVFrame to BGR
sws_scale(
    sws_ctx,
    frame->data, frame->linesize,  // Source data
    0, frame->height,              // Source frame range
    dst_data, dst_linesize         // Destination data
);

Step 2: Detect I-Frames & Generate Edge Maps #

Now that you have a BGR Mat, check if the frame is an I-frame (your existing code) then use OpenCV's Canny edge detector—this is the most common and easy-to-use method for edge detection.

// Check if this is an I-frame (your existing code)
char pictType = av_get_picture_type_char(frame->pict_type);
if (pictType == 'I') {
    // Convert BGR to grayscale (Canny requires a single-channel image)
    cv::Mat gray_frame;
    cv::cvtColor(bgr_frame, gray_frame, cv::COLOR_BGR2GRAY);

    // Optional but recommended: Apply Gaussian blur to reduce noise
    cv::Mat blurred_frame;
    cv::GaussianBlur(gray_frame, blurred_frame, cv::Size(3, 3), 0);

    // Run Canny edge detection
    cv::Mat edge_map;
    int low_threshold = 50;  // Adjust these based on your video's content
    int high_threshold = 150;
    cv::Canny(blurred_frame, edge_map, low_threshold, high_threshold);

    // Do whatever you need with the edge map!
    // Example: Save to disk
    cv::imwrite("latest_i_frame_edge.jpg", edge_map);
    // Or send it over your stream, process it further, etc.
}

Step 3: Clean Up Resources #

Don't forget to free FFmpeg's resources when you're done—OpenCV's Mat will handle its own memory automatically.

// When you're finished with the scaling context (e.g., at program exit)
sws_freeContext(sws_ctx);

// Free the AVFrame after processing (if you allocated it)
av_frame_free(&frame);

Quick Tips for Beginners #

• Linking Libraries: Make sure your project links against both FFmpeg libraries (libswscale, libavutil, etc.) and OpenCV libraries (opencv_core, opencv_imgproc, opencv_imgcodecs).
• Pixel Format Checks: If your frame isn't YUV420P, use av_get_pix_fmt_name(frame->format) to see its format, then adjust the source format in sws_getContext.
• Real-Time Performance: For streaming, avoid imwrite (it's slow) and process the edge map in memory (e.g., encode it back to a stream buffer) instead.

内容的提问来源于stack exchange，提问作者8793