Hey there! Let's break down your two questions about building an RTSP player with FFmpeg and fixing that annoying latency issue— I’ve been there, so I feel your pain.

快速验证：用FFmpeg命令行播放 #

If you just want to test an RTSP stream quickly without writing code, FFmpeg’s built-in ffplay tool is your best friend. Run this command:

ffplay rtsp://your-rtsp-stream-url

It handles all the heavy lifting (streaming, decoding, rendering) out of the box, perfect for checking if your stream is working.

自研播放器的核心流程 #

If you’re building a custom player from scratch, here’s the step-by-step workflow:

• Initialize FFmpeg: Start by setting up the FFmpeg environment. For older versions (<=4.0), call av_register_all() to register all formats, and avformat_network_init() to initialize network support (newer FFmpeg versions don’t require av_register_all() anymore).
• Open the RTSP stream: Use avformat_open_input() to connect to your RTSP URL. Pass in a dictionary of options (we’ll get to latency-related options later) if needed. Then call avformat_find_stream_info() to fetch details about the audio/video streams, and note down the indices of the video and audio streams.
• Set up decoders: For each stream (video/audio), find the corresponding decoder with avcodec_find_decoder(), then open it using avcodec_open2().
• Read, decode, and render: Loop through the stream:
  1. Use av_read_frame() to read packets from the stream.
  2. Send packets to the decoder with avcodec_send_packet(), then receive decoded frames with avcodec_receive_frame().
  3. Convert the decoded video frame (usually YUV format) to a display-friendly format (like RGB) and render it using a GUI framework (SDL, Qt, DirectX, etc.). For audio, send the PCM data to your system’s audio output.
• Clean up: When playback ends, make sure to close decoders, free the format context, and shut down the network module with avformat_network_deinit().

First off: av_seek_frame() won’t help with live streams— it’s designed for seeking in pre-recorded media, not real-time feeds. Let’s go through the actual fixes that will get you down to that 300ms latency like VLC:

• Tweak RTSP transport settings
  By default, FFmpeg might use TCP for RTSP (which is reliable but adds latency). Switch to UDP for lower latency (you might get minor packet loss, but most live streams can handle it). Add these options when opening the stream:

  AVDictionary *opts = NULL;
  // Use UDP for lower latency
  av_dict_set(&opts, "rtsp_transport", "udp", 0);
  // Set a short timeout to avoid hanging on slow streams
  av_dict_set(&opts, "stimeout", "500000", 0); // 500ms timeout
  avformat_open_input(&fmt_ctx, rtsp_url, NULL, &opts);
  

• Cut down on buffer sizes
  FFmpeg’s default buffers are optimized for smooth playback, not low latency. Shrink them to minimize frame accumulation:

  av_dict_set(&opts, "buffer_size", "102400", 0); // Smaller buffer (100KB)
  av_dict_set(&opts, "max_delay", "100000", 0); // Cap max delay at 100ms (in microseconds)
  

  Also, in your custom player, don’t cache too many decoded frames. As soon as you decode a frame, render it— don’t let a queue of unshown frames build up.

• Disable frame reordering and speed up decoding
  B-frames (bidirectional prediction frames) require reordering, which adds latency. Tell the decoder to prioritize speed over quality:

  AVCodecContext *codec_ctx = ...; // Your video codec context
  // Enable fast decoding
  codec_ctx->flags2 |= AV_CODEC_FLAG2_FAST;
  // Skip loop filtering to save processing time
  codec_ctx->skip_loop_filter = AVDISCARD_ALL;
  

  If your stream allows it, ask the RTSP server to send only I/P frames (no B-frames) for even lower latency.

• Adjust sync strategy
  If you’re syncing video to audio, make sure your audio buffer is as small as possible. For pure video streams, skip audio sync entirely— render each frame as soon as it’s decoded, using the system clock as a loose reference instead of waiting for audio.
  Also, set tight sync thresholds: if a frame’s timestamp is behind the current playback time by more than a small margin (like 50ms), drop it instead of waiting to play it.

• Drop old frames to keep up with real time
  VLC’s low-latency mode works by discarding outdated frames to stay in sync with the live stream. In your player, keep track of how many frames are waiting in your decode/render queue. If the queue grows beyond 2-3 frames, drop the oldest ones— this prevents latency from piling up as the stream runs.

• Use hardware acceleration
  If your CPU is struggling to decode frames in time, hardware acceleration (NVDEC, VA-API, etc.) can drastically speed up decoding. FFmpeg supports hardware decoders— you’ll need to initialize a hardware device context and attach it to your codec context when opening the decoder.

内容的提问来源于stack exchange，提问作者geeeek