基于Android CameraX图像分析的实时文档角点检测实现问询

Hey there! Let's walk through how to build real-time document corner detection with CameraX, addressing each of your questions with practical, performant solutions—this is exactly the core logic behind most document scanning apps.

CameraX's ImageAnalysis module is designed for frame-by-frame processing, and here's how to wire it up for document corner detection:

Step 1: Configure the Analyzer #

First, create a custom ImageAnalysis.Analyzer to process each camera frame. The key here is to work efficiently with the YUV_420_888 format (CameraX's default) by using the Y channel directly as grayscale data—no need to convert the entire image to RGB, which saves processing time.

class DocumentCornerAnalyzer(
    private val previewView: PreviewView,
    private val onCornersDetected: (List<PointF>) -> Unit
) : ImageAnalysis.Analyzer {

    override fun analyze(imageProxy: ImageProxy) {
        val image = imageProxy.image ?: run {
            imageProxy.close()
            return
        }

        // Extract Y channel (grayscale) for faster processing
        val yPlane = image.planes[0]
        val yBuffer = yPlane.buffer
        val yData = ByteArray(yBuffer.remaining())
        yBuffer.get(yData)

        val width = image.width
        val height = image.height

        // Convert to OpenCV Mat (we'll use OpenCV for optimized detection)
        val grayMat = Mat(height, width, CvType.CV_8UC1)
        grayMat.put(0, 0, yData)

        // Run detection logic (we'll define this next)
        val rawCorners = detectDocumentCorners(grayMat)

        // Convert image coordinates to preview UI coordinates (critical for accurate drawing)
        val uiCorners = convertToPreviewCoordinates(
            rawCorners,
            imageProxy.imageInfo.rotationDegrees,
            previewView.width,
            previewView.height
        )

        // Pass corners to your UI layer to draw the quadrilateral
        onCornersDetected(uiCorners)

        // Clean up resources
        grayMat.release()
        imageProxy.close()
    }
}

Step 2: Coordinate System Conversion #

CameraX's image frames are often rotated relative to the preview UI. You'll need to adjust detected corner coordinates to match the preview's orientation and scale:

private fun convertToPreviewCoordinates(
    corners: List<PointF>,
    rotationDegrees: Int,
    previewWidth: Int,
    previewHeight: Int
): List<PointF> {
    val matrix = Matrix()

    // Account for camera rotation
    matrix.postRotate(rotationDegrees.toFloat())

    // Scale to match preview view size
    val scaleX = previewWidth.toFloat() / if (rotationDegrees % 180 == 0) corners.first().x else corners.first().y
    val scaleY = previewHeight.toFloat() / if (rotationDegrees % 180 == 0) corners.first().y else corners.first().x
    matrix.postScale(scaleX, scaleY)

    // Apply transformation to all corners
    val transformedCorners = corners.toTypedArray()
    matrix.mapPoints(transformedCorners)

    return transformedCorners.toList()
}

Harris corner detection can be slow on mobile if implemented naively. Here's how to make it real-time:

Use Shi-Tomasi Instead of Harris #

Shi-Tomasi is faster, more robust, and directly returns the strongest N corners (perfect for finding 4 document corners). Pair it with contour detection for even better accuracy:

private fun detectDocumentCorners(grayMat: Mat): List<PointF> {
    // Reduce noise with Gaussian blur
    Imgproc.GaussianBlur(grayMat, grayMat, Size(5.0, 5.0), 0.0)

    // Step 1: Find all strong corners with Shi-Tomasi
    val corners = MatOfPoint2f()
    Imgproc.goodFeaturesToTrack(
        grayMat,
        corners,
        10, // Max 10 corners to filter from
        0.01, // Quality threshold (reject weak corners)
        10.0, // Minimum distance between corners
        Mat(),
        3,
        false, // Disable Harris, use Shi-Tomasi
        0.04
    )

    // Step 2: Find the largest contour (likely the document)
    val edges = Mat()
    Imgproc.Canny(grayMat, edges, 50.0, 150.0)
    val contours = mutableListOf<MatOfPoint>()
    val hierarchy = Mat()
    Imgproc.findContours(edges, contours, hierarchy, Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE)

    val largestContour = contours.maxByOrNull { Imgproc.contourArea(it) } ?: return emptyList()

    // Step 3: Approximate contour to a quadrilateral
    val approx = MatOfPoint2f()
    val contourPerimeter = Imgproc.arcLength(MatOfPoint2f(*largestContour.toArray()), true)
    Imgproc.approxPolyDP(MatOfPoint2f(*largestContour.toArray()), approx, contourPerimeter * 0.02, true)

    // Return 4 corners if approximation worked; fall back to Shi-Tomasi's top 4
    return if (approx.toList().size == 4) {
        approx.toList().map { PointF(it.x, it.y) }
    } else {
        selectOuterFourCorners(corners.toList())
    }
}

private fun selectOuterFourCorners(corners: List<PointF>): List<PointF> {
    if (corners.size < 4) return emptyList()
    // Sort to find the four outer corners (top-left, top-right, bottom-right, bottom-left)
    val sortedBySum = corners.sortedBy { it.x + it.y }
    val topLeft = sortedBySum[0]
    val bottomRight = sortedBySum.last()
    val sortedByDiff = corners.sortedBy { it.x - it.y }
    val topRight = sortedByDiff[0]
    val bottomLeft = sortedByDiff.last()
    return listOf(topLeft, topRight, bottomRight, bottomLeft)
}

Additional Optimizations #

• Downscale the image: Resize the grayscale frame to 640x480 before detection—this cuts down computation time drastically without losing corner accuracy.
• Use OpenCV's optimized builds: OpenCV uses NEON instructions and hardware acceleration on mobile, which is way faster than handwritten Kotlin/Java code.
• Limit detection to ROI: If users typically position documents in the center, only process the middle 80% of the frame to save cycles.

When the document is stationary, re-running detection every frame is wasted effort. Here's how to optimize:

Add Motion Detection with Frame Differencing #

Compare the current frame to the previous one to detect movement. If no motion is detected, reuse the last detected corners:

class DocumentCornerAnalyzer(
    private val previewView: PreviewView,
    private val onCornersDetected: (List<PointF>) -> Unit
) : ImageAnalysis.Analyzer {

    private var previousGrayMat: Mat? = null
    private var cachedCorners: List<PointF> = emptyList()
    private val motionThreshold = 5000 // Adjust based on testing

    override fun analyze(imageProxy: ImageProxy) {
        // ... (previous code to get grayMat)

        // Check for motion between frames
        val hasMotion = previousGrayMat?.let { prevMat ->
            val diff = Mat()
            Core.absdiff(grayMat, prevMat, diff)
            val totalDiff = Core.sumElems(diff)[0]
            totalDiff > motionThreshold
        } ?: true // Always run detection on first frame

        // Use cached corners if no motion
        val currentCorners = if (hasMotion) {
            detectDocumentCorners(grayMat).also { cachedCorners = it }
        } else {
            cachedCorners
        }

        // ... (coordinate conversion and callback)

        // Update previous frame
        previousGrayMat?.release()
        previousGrayMat = grayMat.clone()
    }
}

Smooth Corner Updates #

Even with motion detection, minor jitter can happen. Add a smoothing step by blending new corners with cached ones:

private fun smoothCorners(newCorners: List<PointF>, cachedCorners: List<PointF>): List<PointF> {
    if (cachedCorners.isEmpty()) return newCorners
    return newCorners.zip(cachedCorners).map { (new, old) ->
        PointF(
            new.x * 0.3f + old.x * 0.7f,
            new.y * 0.3f + old.y * 0.7f
        )
    }
}

Call this when updating cached corners to make the quadrilateral movement feel smoother.

内容的提问来源于stack exchange，提问作者Ak01