Android端OpenCV用CAP_MJPEG/CAP_GSTREAMER无法获取ESP32-CAM流帧

我正尝试在Android设备上从ESP32-CAM获取流帧用于目标检测，已知ESP32-CAM输出MJPG流，但使用CAP_MJPEG和CAP_GSTREAMER两种方式均失败，具体情况如下：

1. 直接使用MJPG流尝试 #

使用浏览器可正常访问的IP地址URL（如http://192.168.0.100），代码实现：

var urlWithIpAddress = getUrlWithIpAddressFromGStreamerCommand(streamAddress)
var fullGStreamerCommand = "gst-launch-1.0 " + streamAddress // gst-launch-1.0 souphttpsrc etc etc....
//actual code meant for reading from the gstreamer lib
lifecycleScope.launch {
    var videoCapturing = VideoCapture(urlWithIpAddress, CAP_OPENCV_MJPEG)
    videoCapturing.open(urlWithIpAddress, CAP_OPENCV_MJPEG)
}

Logcat输出日志：

app_function Creating pipeline in CustomData at 0xb40000709a195810
app_function custom pipeline string: souphttpsrc location=http://192.168.0.100 ! multipartdemux ! jpegdec ! videoconvert ! autovideosink
urlWithIpAddress is http://192.168.0.100
[ WARN:0@26.437] global cap.cpp:206 open VIDEOIO(CV_MJPEG): backend is generally available but can't be used to capture by name
[ WARN:0@26.438] global cap.cpp:206 open VIDEOIO(CV_MJPEG): backend is generally available but can't be used to capture by name

2. 结合GStreamer尝试 #

参考资料显示OpenCV需搭配GStreamer/FFmpeg才能获取流，使用的GStreamer管道命令：

souphttpsrc location=http://192.168.0.102 ! multipartdemux ! jpegdec ! videoconvert ! autovideosink 

添加gst-launch-1.0前缀并指定CAP_GSTREAMER仍无效，代码实现：

var urlWithIpAddress = getUrlWithIpAddressFromGStreamerCommand(streamAddress)
var fullGStreamerCommand = "gst-launch-1.0 " + streamAddress // gst-launch-1.0 souphttpsrc etc etc....
//actual code meant for reading from the gstreamer lib
lifecycleScope.launch {
    var videoCapturing = VideoCapture(fullGStreamerCommand, CAP_GSTREAMER)
    videoCapturing.open(fullGStreamerCommand, CAP_CAP_GSTREAMER)
}

日志显示：

urlWithIpAddress is http://192.168.0.100
videoCapturing opening:  gst-launch-1.0 souphttpsrc location=http://192.168.0.100 ! multipartdemux ! jpegdec ! videoconvert ! autovideosink 
videoCapturing isOpened:  false 
videoCapturing isOpened():  false 
Video capture failed, no frames obtained. videoCapturing.isOpened returned false. 
video capture releasing!  

额外偶现错误 #

OpenCV(4.8.0) Error: Requested object was not found (could not open directory: /data/app/~~h0G0FSTdUBM1n2pgU7YM8g==/com.pkgname.streamplayer-fI_rhcyXiIM5ZinT8OdKFg==/base.apk!/lib/x86_64) in glob_rec, file /home/ci/opencv/modules/core/src/glob.cpp, line 279

可行解决方案及替代方案 #

• 修正GStreamer管道格式：OpenCV的GStreamer后端不需要gst-launch-1.0前缀，且需将输出端改为appsink（而非autovideosink），因为OpenCV需要从appsink获取帧数据。同时添加is-live=true适配实时流，指定输出格式为BGR（OpenCV默认格式）：

  souphttpsrc location=http://192.168.0.100 is-live=true ! multipartdemux ! jpegdec ! videoconvert ! video/x-raw,format=BGR ! appsink drop=true sync=false
  

  对应代码实现：

  lifecycleScope.launch {
      val gstPipeline = "souphttpsrc location=http://192.168.0.100 is-live=true ! multipartdemux ! jpegdec ! videoconvert ! video/x-raw,format=BGR ! appsink drop=true sync=false"
      val videoCapturing = VideoCapture()
      if (videoCapturing.open(gstPipeline, CAP_GSTREAMER)) {
          var frame = Mat()
          while (videoCapturing.read(frame)) {
              // 在此处理帧（如目标检测）
          }
          videoCapturing.release()
          frame.release()
      } else {
          Log.e("StreamCapture", "无法打开GStreamer管道")
      }
  }
  

• 验证OpenCV编译配置：确保Android平台的OpenCV库编译时启用了GStreamer支持。多数预编译的OpenCV Android包默认未开启该选项，需自行编译OpenCV并勾选WITH_GSTREAMER，或使用带GStreamer支持的自定义构建包。

• 替代方案：HTTP直接拉取单帧：绕过OpenCV视频捕获后端，通过HTTP请求直接拉取MJPG流的单帧，解码为OpenCV的Mat对象：

  suspend fun fetchEsp32Frame(url: String): Mat? {
      return try {
          val client = OkHttpClient()
          val request = Request.Builder().url(url).build()
          val response = client.newCall(request).execute()
          val inputStream = response.body?.byteStream() ?: return null
          val bitmap = BitmapFactory.decodeStream(inputStream)
          val mat = Mat()
          Utils.bitmapToMat(bitmap, mat)
          Core.cvtColor(mat, mat, COLOR_RGBA2BGR) // 转换为OpenCV常用的BGR格式
          mat
      } catch (e: Exception) {
          e.printStackTrace()
          null
      }
  }
  
  // 使用示例
  lifecycleScope.launch {
      while (true) {
          val frame = fetchEsp32Frame("http://192.168.0.100/capture") // ESP32-CAM通常提供单独的帧捕获接口
          frame?.let {
              // 目标检测处理逻辑
              it.release()
          }
          delay(30) // 控制帧率
      }
  }
  

• 修复库加载路径错误：针对偶现的x86_64目录找不到错误，在app/build.gradle中配置ABI过滤，只保留设备支持的架构：

  android {
      defaultConfig {
          ndk {
              abiFilters 'arm64-v8a', 'armeabi-v7a'
          }
      }
  }
  

内容的提问来源于stack exchange，提问作者fatbringer