Android CameraX视频录制与实时人脸眨眼检测性能优化问询

核心优化思路 #

放弃PreviewView.getBitmap()的低效帧获取方式，改用VideoCapture原生帧回调获取数据，结合MLKit人脸关键点实现无分类模式的眨眼检测，再通过线程优化和帧采样保证目标帧率。

1. 改用VideoCapture帧回调获取原始帧 #

VideoCapture在录制过程中可以直接输出未编码的帧数据，无需从PreviewView截图，彻底避免主线程的bitmap生成耗时。

修改相机绑定代码，给videoCapture添加帧回调：

private void bindCameraUseCases() {
    try {
        Preview.Builder builder = new Preview.Builder();
        previewUseCase = builder.build();
        previewUseCase.setSurfaceProvider(fragmentCameraBinding.previewView.getSurfaceProvider());
        
        Recorder recorder = new Recorder.Builder()
                .setQualitySelector(QualitySelector.from(Quality.LOWEST, FallbackStrategy.higherQualityOrLowerThan(Quality.LOWEST)))
                .build();
        videoCapture = VideoCapture.withOutput(recorder);

        // 创建后台线程池处理帧检测
        ExecutorService detectionExecutor = Executors.newFixedThreadPool(2);
        // 添加VideoCapture帧回调
        videoCapture.setOnFrameCapturedCallback(detectionExecutor, new VideoCapture.OnFrameCapturedCallback() {
            @Override
            public void onFrameCaptured(@NonNull VideoCapture.Output output, @NonNull Frame frame, @NonNull CaptureResult captureResult) {
                super.onFrameCaptured(output, frame, captureResult);
                // 在这里处理帧数据，转成MLKit可用格式
                processFrameForFaceDetection(frame);
            }
        });

        new ViewModelProvider(this, ViewModelProvider.AndroidViewModelFactory.getInstance(getActivity().getApplication()))
                .get(CameraXViewModel.class)
                .getProcessCameraProvider()
                .observe(getViewLifecycleOwner(), provider -> {
                    cameraProvider = provider;
                    try {
                        cameraProvider.unbindAll();
                        cameraProvider.bindToLifecycle(getViewLifecycleOwner(), cameraSelector, previewUseCase, videoCapture);
                    } catch (Exception e) {
                        e.printStackTrace();
                    }
                });
    } catch (Exception e) {
        e.printStackTrace();
    }
}

2. 无CLASSIFICATION_MODE_ALL的眨眼检测实现 #

通过MLKit的人脸关键点计算眼部纵横比（EAR）来判断眨眼，无需启用分类模式，保证检测速度。

首先配置高效的FaceDetectorOptions：

private FaceDetector createFastFaceDetector() {
    FaceDetectorOptions options = new FaceDetectorOptions.Builder()
            .setPerformanceMode(FaceDetectorOptions.PERFORMANCE_MODE_FAST)
            .setLandmarkMode(FaceDetectorOptions.LANDMARK_MODE_ALL) // 启用眼部关键点
            .setContourMode(FaceDetectorOptions.CONTOUR_MODE_ALL) // 启用轮廓提升关键点精度
            .setClassificationMode(FaceDetectorOptions.CLASSIFICATION_MODE_NONE) // 关闭分类模式提速
            .build();
    return FaceDetection.getClient(options);
}

然后实现EAR计算和眨眼判断逻辑：

private void processFrameForFaceDetection(Frame frame) {
    // 将CameraX的Frame转成MLKit的InputImage
    InputImage image = InputImage.fromMediaImage(frame.getImage(), frame.getImageInfo().getRotationDegrees());
    
    // 调用人脸检测
    createFastFaceDetector().process(image)
            .addOnSuccessListener(faces -> {
                for (Face face : faces) {
                    // 计算左右眼EAR值
                    float leftEar = calculateEyeAspectRatio(face.getLandmarks().get(FaceLandmark.LEFT_EYE));
                    float rightEar = calculateEyeAspectRatio(face.getLandmarks().get(FaceLandmark.RIGHT_EYE));
                    float avgEar = (leftEar + rightEar) / 2;
                    
                    // EAR阈值设为0.2，低于则判定为眨眼
                    if (avgEar < 0.2) {
                        onBlinkDetected();
                    }
                }
            })
            .addOnFailureListener(e -> e.printStackTrace());
}

private float calculateEyeAspectRatio(FaceLandmark eyeLandmark) {
    if (eyeLandmark == null) return 1.0f;
    
    List<PointF> eyePoints = eyeLandmark.getPosition();
    if (eyePoints.size() < 6) return 1.0f;
    
    // 计算垂直距离之和
    float verticalSum = distance(eyePoints.get(1), eyePoints.get(5)) + distance(eyePoints.get(2), eyePoints.get(4));
    // 计算水平距离
    float horizontalDistance = distance(eyePoints.get(0), eyePoints.get(3));
    
    return verticalSum / (2 * horizontalDistance);
}

private float distance(PointF p1, PointF p2) {
    float dx = p1.x - p2.x;
    float dy = p1.y - p2.y;
    return (float) Math.sqrt(dx*dx + dy*dy);
}

private void onBlinkDetected() {
    // 处理眨眼事件，更新UI需切换到主线程
    requireActivity().runOnUiThread(() -> {
        // 示例：更新UI提示眨眼
    });
}

3. 帧采样优化，控制处理帧率 #

针对24-25fps的原始帧率，通过固定间隔采样保证稳定在16-18fps：

private long lastProcessTime = 0;
private static final long PROCESS_INTERVAL = 55; // 约18fps（1000/18≈55）

@Override
public void onFrameCaptured(@NonNull VideoCapture.Output output, @NonNull Frame frame, @NonNull CaptureResult captureResult) {
    super.onFrameCaptured(output, frame, captureResult);
    
    long currentTime = System.currentTimeMillis();
    if (currentTime - lastProcessTime >= PROCESS_INTERVAL) {
        lastProcessTime = currentTime;
        processFrameForFaceDetection(frame);
    }
}

4. 资源回收优化 #

在Fragment销毁时关闭线程池，避免资源泄漏：

private ExecutorService detectionExecutor;

// 在bindCameraUseCases中初始化线程池
detectionExecutor = Executors.newFixedThreadPool(2);

@Override
public void onDestroy() {
    super.onDestroy();
    if (detectionExecutor != null) {
        detectionExecutor.shutdown();
    }
}

内容的提问来源于stack exchange，提问作者Faizan Ahmad