Android Native中sws_scale处理RGB0帧耗时过长及FFmpeg帧缩放性能问题问询

Hey there, let's dig into your performance issues with FFmpeg frame processing on Android's Native layer. I've dealt with similar problems in low-latency video pipelines, so here's a breakdown of what's happening and how to fix it:

1. Why Scaling to Surface Size Takes 25-30ms (vs 1ms for Camera Output Size) #

The massive time difference almost always boils down to two core factors:

• Size/Aspect Ratio Disparity: If your Surface's resolution or aspect ratio is drastically different from the original video frame (e.g., scaling 1080p to a non-standard Surface size, or upscaling/downscaling by a large factor), sws_scale has to do far more interpolation work. When scaling to the camera output size, it’s likely the dimensions are nearly identical to the source frame—so FFmpeg’s scaling code is doing minimal work (or even just copying data, hence the 1ms time).
• Software vs Hardware Processing: Basic sws_scale and related calls are software-based. For cross-resolution scaling, software interpolation is inherently slow compared to hardware acceleration.

Fixes for Surface Scaling Latency: #

• Use Hardware-Accelerated Scaling: On Android, leverage OpenGL ES or MediaCodec to handle scaling directly on the GPU. This can cut scaling time down to almost nothing, even for big resolution jumps. Bind the Surface to an OpenGL texture, scale it with shader code, and render directly—no need for FFmpeg software scaling.
• Optimize swsContext Initialization:
  • Pick a speed-focused scaling algorithm like SWS_FAST_BILINEAR instead of slower defaults like SWS_BILINEAR or SWS_LANCZOS.
  • Reuse the same SwsContext for all scaling operations—don’t recreate it every frame. Initializing the context has overhead that adds up quickly.
    Example optimized initialization:

  struct SwsContext* sws_ctx = sws_getContext(
      src_width, src_height, src_pix_fmt,
      surface_width, surface_height, dst_pix_fmt,
      SWS_FAST_BILINEAR, // Prioritize speed over fine-grained quality
      NULL, NULL, NULL
  );
  

• Match Aspect Ratio Where Possible: If your Surface allows it, adjust its size to match the source video's aspect ratio. This reduces the amount of interpolation needed (no stretching/squeezing pixels unnecessarily).

2. Long sws_scale Times for RGB0 Frames Causing Delay #

RGB0 is a 32-bit pixel format (4 bytes per pixel), which means it has 2.6x more data to process than YUV420P (1.5 bytes per pixel). Add in YUV-to-RGB color space conversion, and software processing gets slow fast.

Fixes for RGB0 Processing Latency: #

• Skip YUV-to-RGB Conversion Entirely: Most Android Surfaces support YUV_420_888 format directly. Instead of converting to RGB0, pass the raw YUV420P frames to the Surface. This eliminates the color conversion overhead entirely—this is the biggest win for low latency.
• Optimize Color Conversion in swsContext: If you must convert to RGB0, explicitly set color space parameters to avoid unnecessary conversions. For example, if your source uses BT.601 and the target expects BT.709, specify these in the sws_getContext call:

  struct SwsContext* sws_ctx = sws_getContext(
      src_w, src_h, src_fmt,
      dst_w, dst_h, AV_PIX_FMT_RGB0,
      SWS_FAST_BILINEAR,
      NULL, NULL, NULL
  );
  // Explicitly define color space mappings
  sws_setColorspaceDetails(sws_ctx,
      sws_getCoefficients(SWS_CS_ITU601),
      sws_getCoefficients(SWS_CS_ITU709),
      0, 0, 0, 0, 0);
  

• Use Hardware-Accelerated Color Conversion: Again, OpenGL ES can handle YUV-to-RGB conversion in the GPU with a simple shader. Bind the YUV planes as textures, sample them in the shader, and output RGB directly to the Surface—this is way faster than software sws_scale.

Quick Recap #

The biggest performance gains come from moving scaling and color conversion to hardware (GPU/MediaCodec) instead of relying on FFmpeg's software tools. Reusing SwsContext instances and choosing speed-focused algorithms also helps reduce overhead for software-based workflows.

内容的提问来源于stack exchange，提问作者AJit