使用onPreviewFrame运行机器学习模型的问题排查

Hey there, let's work through your problem with the Legacy Camera API's onPreviewFrame callback and ML Kit. I see you're running into two main headaches: null Bitmaps from raw YUV data, and performance warnings when converting to JPEG/Bitmap. Let's break down what's happening and fix it step by step.

First, Why Your First Approach Returns Null #

Your first code tries to use yuv.getYuvData() and pass that directly to BitmapFactory.decodeByteArray()—that's the root cause! BitmapFactory can't decode raw YUV byte arrays; it only understands compressed formats like JPEG or PNG. The raw YUV data isn't a valid image format for BitmapFactory, so it returns null every time.

Second, Why the Performance Warning Happens #

Your second approach converts YUV to JPEG, then to Bitmap, which is super inefficient. ML Kit explicitly states it supports NV21 byte arrays directly (which is exactly what Legacy Camera gives you, since you confirmed parameters.getPreviewFormat() returns 17—aka ImageFormat.NV21). Converting to Bitmap adds unnecessary processing overhead, hence the warning.

Also, when you tried switching to ImageFormat.YUV_420_888, you got an error because the Legacy Camera API doesn't support that format—it's exclusive to Camera2. So sticking with NV21 is the right move here.

The Optimal Solution: Pass NV21 Directly to ML Kit #

Instead of converting to Bitmap at all, use ML Kit's built-in support for NV21. Here's how to adjust your code:

Step 1: Calculate Camera Rotation Angle #

ML Kit needs to know the correct rotation of the preview frame (since camera orientation doesn't always match screen orientation). Add this helper method:

private int getCameraRotationAngle(Camera camera) {
    Display display = ((WindowManager) getSystemService(Context.WINDOW_SERVICE)).getDefaultDisplay();
    int rotation = display.getRotation();
    int degrees = 0;
    switch (rotation) {
        case Surface.ROTATION_0: degrees = 0; break;
        case Surface.ROTATION_90: degrees = 90; break;
        case Surface.ROTATION_180: degrees = 180; break;
        case Surface.ROTATION_270: degrees = 270; break;
    }

    Camera.CameraInfo info = new Camera.CameraInfo();
    Camera.getCameraInfo(Camera.CameraInfo.CAMERA_FACING_BACK, info);
    int result;
    if (info.facing == Camera.CameraInfo.CAMERA_FACING_FRONT) {
        result = (info.orientation + degrees) % 360;
        result = (360 - result) % 360;  // Compensate for front camera mirroring
    } else {  // Back-facing camera
        result = (info.orientation - degrees + 360) % 360;
    }
    return result;
}

Step 2: Update onPreviewFrame to Use NV21 Directly #

Modify your callback to skip Bitmap conversion entirely:

@Override 
public void onPreviewFrame(byte[] data, Camera camera) { 
    Camera.Parameters parameters = camera.getParameters(); 
    Camera.Size previewSize = parameters.getPreviewSize();
    int width = previewSize.width;
    int height = previewSize.height;
    int rotation = getCameraRotationAngle(camera);

    // Create FirebaseVisionImage directly from NV21 byte array
    FirebaseVisionImage image = FirebaseVisionImage.fromByteArray(
        data,
        width,
        height,
        rotation,
        FirebaseVisionImageFormat.NV21
    );

    // Pass the image to your barcode extraction method
    extractBarcode(image, null); // If you don't need the Bitmap for display, pass null
}

This eliminates the performance warning and skips all the unnecessary conversion steps. ML Kit will process the raw NV21 data efficiently.

If You Do Need a Bitmap (e.g., for Preview Display) #

If you need to show the preview frame as a Bitmap, avoid converting to JPEG. Use RenderScript for a much faster YUV-to-Bitmap conversion:

private Bitmap convertNV21ToBitmap(byte[] nv21Data, int width, int height) {
    RenderScript rs = RenderScript.create(this);
    ScriptIntrinsicYuvToRGB yuvToRgbIntrinsic = ScriptIntrinsicYuvToRGB.create(rs, Element.U8_4(rs));

    Allocation inAllocation = Allocation.createSized(rs, Element.U8(rs), nv21Data.length);
    Allocation outAllocation = Allocation.createTyped(rs, new Type.Builder(rs, Element.U8_4(rs))
            .setX(width).setY(height).create());

    inAllocation.copyFrom(nv21Data);
    yuvToRgbIntrinsic.setInput(inAllocation);
    yuvToRgbIntrinsic.forEach(outAllocation);

    Bitmap bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
    outAllocation.copyTo(bitmap);

    // Clean up RenderScript resources
    inAllocation.destroy();
    outAllocation.destroy();
    yuvToRgbIntrinsic.destroy();
    rs.destroy();

    return bitmap;
}

You can call this method if you need to display the frame, but still pass the raw NV21 data to ML Kit for processing.

Quick Recap #

• Null Bitmap issue: You were passing raw YUV data to BitmapFactory, which can't decode it. Use either RenderScript to convert to Bitmap or skip Bitmap entirely for ML Kit.
• Performance warning: Converting to Bitmap is unnecessary—ML Kit supports NV21 directly.
• YUV_420_888 error: Legacy Camera doesn't support this format; stick with NV21.

内容的提问来源于stack exchange，提问作者insta catering