升级FFmpeg从1.1到3.3后H264 RTSP流解码无帧输出问题

Hey there, let's break down why your H.264 RTSP decoding stopped working when moving from FFmpeg 1.1 to 3.3—this is a super common snag with FFmpeg's evolving API, and we'll get you back on track quickly.

First, to recap your issue: using the old avcodec_decode_video2 function, you’re seeing got_picture stuck at 0, the function returns the full input packet size (so it’s consuming data), but no decoded frames ever come through. Here’s what’s going wrong and how to fix it:

1. The Old Decoding API Has Compatibility Gaps in 3.x #

FFmpeg 3.x shifted hard toward the new send/receive decoding workflow (av_codec_send_packet + av_codec_receive_frame), and the legacy avcodec_decode_video2 function got internal changes that make it less reliable for streaming H.264. The old API was built around a "one packet, one frame" assumption, but H.264 often needs multiple packets to build a single frame (or one packet can hold multiple frames)—the new API handles this streaming-specific logic way better.

Switch to the New API (Recommended Fix) #

Replace your avcodec_decode_video2 call with this pattern—it’s what FFmpeg officially recommends for 3.1+ versions:

// Send the packet to the decoder first
int ret = av_codec_send_packet(codec_ctx, &packet);
if (ret < 0) {
    // Handle invalid packet errors
    av_packet_unref(&packet);
    return ret;
}

// Pull all available frames from the decoder
while (ret >= 0) {
    ret = av_codec_receive_frame(codec_ctx, frame);
    if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF) {
        // No more frames right now—need to feed more packets
        break;
    } else if (ret < 0) {
        // Handle decoding errors (corrupted data, etc.)
        break;
    }

    // ✅ You’ve got a valid frame here! Process it as needed
    // Don’t forget to free frame resources when done
    av_frame_unref(frame);
}

// Always clean up the packet after processing
av_packet_unref(&packet);

This API is purpose-built for continuous streams like RTSP, so it’ll handle H.264’s frame-building logic far more reliably than the old function.

2. Decoder Context Initialization Got Strict #

FFmpeg 3.x changed how decoder contexts are set up—you can’t manually set width, height, and other parameters like you could in 1.1 anymore. You need to pull metadata directly from the stream’s codecpar structure to ensure the decoder has all the critical info (like SPS/PPS data, profile, and level) to decode H.264 properly.

Fix Your Decoder Setup #

Update your context initialization to follow this 3.x-compliant pattern:

// Grab the stream's codec parameters
AVCodecParameters *codec_par = stream->codecpar;

// Find the H.264 decoder
AVCodec *codec = avcodec_find_decoder(codec_par->codec_id);
if (!codec) {
    // Handle error: decoder not found
    return -1;
}

// Allocate a codec context
AVCodecContext *codec_ctx = avcodec_alloc_context3(codec);
if (!codec_ctx) {
    // Handle error: out of memory
    return -1;
}

// Copy parameters from the stream to the decoder context (critical step!)
if (avcodec_parameters_to_context(codec_ctx, codec_par) < 0) {
    avcodec_free_context(&codec_ctx);
    return -1;
}

// Open the decoder
if (avcodec_open2(codec_ctx, codec, NULL) < 0) {
    avcodec_free_context(&codec_ctx);
    return -1;
}

Skipping the avcodec_parameters_to_context step was common in 1.1, but 3.x requires it to avoid missing critical H.264 metadata that the decoder needs.

3. Double-Check Packet Hygiene #

Even if you stick with the old API, make sure you’re:

• Calling av_packet_unref on every packet after processing it—leaving packets unref’d can cause state corruption in the decoder.
• Verifying your RTSP demuxer is correctly assembling RTP packets into full H.264 NAL units. FFmpeg’s RTSP demuxer should handle this, but avoid modifying packet data (like stripping start codes) before feeding it to the decoder.

Why This Happened #

FFmpeg 3.3 tightened up decoder state management and input handling. The old avcodec_decode_video2 function was kept for backward compatibility, but it’s now a wrapper around the new send/receive API—and that wrapper doesn’t handle streaming edge cases (like partial frames or multiple NAL units per packet) as smoothly as using the new API directly.

内容的提问来源于stack exchange，提问作者M.Mahdipour