CameraX自定义OpenGL视频管线迁移实现问题求助

背景 #

我现有一个基于Camera2的应用，为解决部分Android设备（如三星设备）的崩溃问题，计划迁移至CameraX。目前我有一个自定义VideoPipeline，流程如下：

• 通过ImageReader获取摄像头帧的CPU访问权限并执行人脸检测
• 检测到人脸后，将帧转发至OpenGL管线（通过ImageWriter或传递HardwareBuffer作为OpenGL纹理）
• 在OpenGL管线中使用OpenGL绘制人脸框
• 绘制完成后，将内容渲染至两个输出Surface：
  • Preview View Surface
  • 录制MP4时的MediaRecorder/Encoder Surface

在Camera2中，我只需将ImageReader的surface附加到摄像头，它就会按ImageReader配置的格式（CPU算法用ImageFormat.YUV_420_888，GPU算法用ImageFormat.PRIVATE）向其流传输帧。

当前实现代码 #

VideoPipeline类 #

class VideoPipeline(
  private val format = ImageFormat.YUV_420_888, // or ImageFormat.PRIVATE
  private val callback: CameraSession.Callback
) : VideoOutput, Closeable {
  companion object {
    private const val MAX_IMAGES = 3
    private const val TAG = "VideoPipeline"
  }

  // Output
  // TODO: Use `Recording`/`Recorder` output from CameraX?

  // TODO: When I didn't override getMediaSpec, CameraX crashed.
  @SuppressLint("RestrictedApi")
  override fun getMediaSpec(): Observable<MediaSpec> {
    val mediaSpec = MediaSpec.builder().setOutputFormat(MediaSpec.OUTPUT_FORMAT_MPEG_4).configureVideo { video ->
      // TODO: Instead of hardcoding that, can I dynamically get those values from the Camera?
      video.setFrameRate(Range(0, 60))
      video.setQualitySelector(QualitySelector.from(Quality.HD))
    }
    return ConstantObservable.withValue(mediaSpec.build())
  }

  @SuppressLint("RestrictedApi")
  override fun onSurfaceRequested(request: SurfaceRequest) {
    val size = request.resolution
    val surfaceSpec = request.deferrableSurface
    Log.i(TAG, "Creating $size Surface... (${request.expectedFrameRate.upper} FPS, expected format: ${surfaceSpec.prescribedStreamFormat}, ${request.dynamicRange})")

    // Create ImageReader
    val imageReader = ImageReader.newInstance(size.width, size.height, format, MAX_IMAGES)

    imageReader.setOnImageAvailableListener({ reader ->
      val image = reader.acquireNextImage() ?: return@setOnImageAvailableListener

      // 1. Detect Faces
      val faces = callback.detectFaces(image)

      // 2. Forward to OpenGL
      onFrame(image)
    }, CameraQueues.videoQueue.handler)

    // Pass the ImageReader surface to CameraX when bound to a lifecycle
    request.provideSurface(imageReader.surface, CameraQueues.videoQueue.executor) { result ->
      imageReader.close()
    }
  }
}

摄像头初始化代码 #

val fpsRange = Range(30, 30)

val preview = Preview.Builder().build()
preview.setSurfaceProvider(previewView.surfaceProvider)

val videoPipeline = VideoPipeline(ImageFormat.YUV_420_888, callback)
val video = VideoCapture.Builder(videoPipeline).also { video ->
  video.setMirrorMode(MirrorMode.MIRROR_MODE_ON_FRONT_ONLY)
  video.setTargetFrameRate(fpsRange)
}.build()

val camera = provider.bindToLifecycle(this, cameraSelector, preview, video)

崩溃错误信息 #

java.lang.UnsupportedOperationException: The producer output buffer format 0x1 doesn't match the ImageReader's configured buffer format 0x23.
  at android.media.ImageReader.nativeImageSetup(Native Method)
  at android.media.ImageReader.acquireNextSurfaceImage(ImageReader.java:439)
  at android.media.ImageReader.acquireNextImage(ImageReader.java:493)
  at com.mrousavy.camera.core.VideoPipeline.onSurfaceRequested$lambda$1(VideoPipeline.kt:99)
  at com.mrousavy.camera.core.VideoPipeline.$r8$lambda$5heGRS9RL_Z-x-BiU-ziCBMS68U(Unknown Source:0)
  at com.mrousavy.camera.core.VideoPipeline$$ExternalSyntheticLambda1.onImageAvailable(Unknown Source:2)
  at android.media.ImageReader$ListenerHandler.handleMessage(ImageReader.java:800)
  at android.os.Handler.dispatchMessage(Handler.java:112)
  at android.os.Looper.loop(Looper.java:216)
  at android.os.HandlerThread.run(HandlerThread.java:65)

问题 #

1. 应该继承VideoOutput并在VideoCapture.Builder中使用，还是直接继承UseCase自行实现所有逻辑？
2. 是否可以在自定义VideoPipeline中复用CameraX的Recorder/Recording实例，将OpenGL输出流传输至其Surface，避免重写MediaMuxer/MediaEncoder相关逻辑？

回答 #

问题1：继承VideoOutput vs 自定义UseCase #

优先选择继承VideoOutput并配合VideoCapture使用，原因如下：

• VideoCapture已经封装了CameraX中视频流的核心逻辑，比如帧率同步、格式协商、生命周期绑定等，自定义UseCase需要从零实现这些底层细节，容易引入兼容性问题，反而违背了迁移到CameraX解决设备崩溃的初衷。
• 你当前的崩溃本质是格式不匹配：CameraX的VideoCapture默认会根据设备能力选择最优输出格式（错误中的0x1是ImageFormat.RGB_565），而你硬指定了ImageFormat.YUV_420_888（0x23），导致格式冲突。解决这个问题的正确方式是在onSurfaceRequested中使用request.deferrableSurface.prescribedStreamFormat创建ImageReader，而不是硬编码格式：

  val imageReader = ImageReader.newInstance(size.width, size.height, surfaceSpec.prescribedStreamFormat, MAX_IMAGES)
  

  这样就能保证ImageReader的格式和CameraX提供的流格式一致，解决崩溃。

问题2：复用CameraX的Recorder/Recording #

完全可以复用，步骤如下：

1. 在VideoPipeline中持有Recorder实例的引用，或者通过构造方法传入已配置好的Recorder。
2. 当需要开始录制时，调用Recorder.startRecording()获取Recording实例，然后获取其对应的Surface。
3. 将这个Surface添加到你的OpenGL管线的输出目标中，和预览Surface一起接收渲染后的帧。
4. 录制控制（开始/暂停/停止）直接通过Recording的API完成，无需自己处理MediaMuxer或MediaEncoder。

示例代码片段：

class VideoPipeline(
  private val callback: CameraSession.Callback,
  private val recorder: Recorder
) : VideoOutput, Closeable {
  private var recording: Recording? = null
  private var recordingSurface: Surface? = null

  fun startRecording(outputFile: File, context: Context) {
    recording = recorder.startRecording(
      FileOutputOptions.Builder(outputFile).build(),
      ContextCompat.getMainExecutor(context),
      object : VideoRecordEventListener {
        override fun onRecordingStarted(event: VideoRecordEvent.Start) {}
        override fun onRecordingStopped(event: VideoRecordEvent.Stop) {
          recordingSurface?.release()
          recordingSurface = null
          recording = null
        }
      }
    )
    recordingSurface = recording?.surface
    // 将recordingSurface添加到OpenGL管线的输出列表中
    addRenderOutputSurface(recordingSurface)
  }

  // ... 其他原有逻辑
}

另外，关于getMediaSpec()的实现，不需要硬编码帧率和分辨率，应该基于SurfaceRequest中的信息动态构建，保证和CameraX实际配置匹配：

@SuppressLint("RestrictedApi")
override fun getMediaSpec(): Observable<MediaSpec> {
  return Observable.create { emitter ->
    val mediaSpec = MediaSpec.builder()
      .setOutputFormat(MediaSpec.OUTPUT_FORMAT_MPEG_4)
      .configureVideo { video ->
        video.setFrameRate(request.expectedFrameRate)
        video.setResolution(request.resolution)
      }
      .build()
    emitter.onNext(mediaSpec)
    emitter.onComplete()
  }
}

内容的提问来源于stack exchange，提问作者mrousavy