Android集成TensorFlow Lite时Buffer尺寸不足的像素处理问题

我在Android Studio中使用Java开发集成TensorFlow Lite的应用，每次拍照后将图像传入模型时，都会出现**"Buffer size is not large enough for pixels"**错误。尝试过将照片转JPEG格式、修改Buffer尺寸计算方式、调整pixelStride和rowStride逻辑、重新处理Bitmap尺寸、修改行填充计算等方法，均未解决问题。

相关代码如下：

@androidx.camera.core.ExperimentalGetImage
private void takeAndAnalyzeImage() {
    if (imageCapture != null) {
        imageCapture.takePicture(cameraExecutor, new ImageCapture.OnImageCapturedCallback() {
            @Override
            public void onCaptureSuccess(ImageProxy image) {
                super.onCaptureSuccess(image);
                Log.d("CameraXApp", "Image capture success. Processing image...");
                Image mediaImage = image.getImage();
                if (mediaImage != null) {
                    Log.d("CameraXApp", "Captured image received. Dimensions: " + mediaImage.getWidth() + "x" + mediaImage.getHeight());
                    try {
                        Log.d("CameraXApp", "Converting to Bitmap...");
                        Bitmap bitmap = toBitmap(mediaImage);
                        if (bitmap != null) { // Check if bitmap is not null
                            Log.d("CameraXApp", "Bitmap conversion successful. Resizing...");
                            // Resize the bitmap to fit the model input size (224x224)
                            Bitmap resizedBitmap = Bitmap.createScaledBitmap(bitmap, 224, 224, true);
                            TensorImage tensorImage = TensorImage.fromBitmap(resizedBitmap);
                            runInference(tensorImage);
                        } else {
                            Log.e("CameraXApp", "Bitmap is null. Cannot resize image.");
                        }
                    } catch (Exception e) {
                        Log.e("CameraXApp", "Error converting image to Bitmap: " + e.getMessage());
                    }
                }
                image.close();
            }

            @Override
            public void onError(ImageCaptureException exception) {
                Log.e("CameraXApp", "Photo capture failed: " + exception.getMessage());
            }
        });
    } else {
        Log.e("CameraXApp", "imageCapture is null. Cannot capture image.");
    }
}

private Bitmap toBitmap(Image image) {
    try {
        ByteBuffer buffer = image.getPlanes()[0].getBuffer();
        int width = image.getWidth();
        int height = image.getHeight();
        int pixelStride = image.getPlanes()[0].getPixelStride();
        int rowStride = image.getPlanes()[0].getRowStride();
        int pixelFormat = image.getFormat();

        // Calculate the expected size based on image dimensions and pixel format
        int expectedSize;
        switch (pixelFormat) {
            case ImageFormat.YUV_420_888:
                expectedSize = width * height * 3 / 2; // YUV_420_888 format
                break;
            case ImageFormat.JPEG:
                expectedSize = buffer.capacity(); // JPEG format
                break;
            default:
                expectedSize = width * height * 4; // Default to ARGB_8888 format
                break;
        }

        Log.d("CameraXApp", "Buffer size: " + buffer.capacity() + ", Expected size: " + expectedSize);

        // Ensure the buffer size is large enough for the pixels
        if (buffer.capacity() < expectedSize) {
            Log.e("CameraXApp", "Buffer not large enough for pixels");
            return null;
        }

        // Check if pixelStride is zero to avoid divide by zero error
        int adjustedWidth = pixelStride != 0 ? width + (rowStride / pixelStride - 1) : width;
        Bitmap bitmap = Bitmap.createBitmap(adjustedWidth, height, Bitmap.Config.ARGB_8888);
        bitmap.copyPixelsFromBuffer(buffer);
        return bitmap;
    } catch (Exception e) {
        Log.e("CameraXApp", "Error converting image to Bitmap: " + e.getMessage());
        return null;
    }
}

private void runInference(TensorImage tensorImage) {
    try {
        FinalModel.Outputs outputs = model.process(tensorImage.getTensorBuffer());
        TensorBuffer outputFeature0 = outputs.getOutputFeature0AsTensorBuffer();
        // Process the output as needed

        // Add a log message to indicate successful analysis
        Log.d("ImageAnalysis", "Image analysis completed successfully");

    } catch (Exception e) {
        Log.e("Error", "Error running inference: " + e.getMessage());
    }
}

问题根源 #

1. YUV_420_888格式处理错误：CameraX默认返回的Image是YUV_420_888格式，该格式包含3个独立平面（Y、U、V），你只取了第一个Y平面的Buffer，却用三个平面的总容量（width*height*3/2）做检查，单个平面的Buffer容量自然远小于总容量，导致错误触发。
2. Bitmap转换逻辑错误：直接将Y平面的Buffer复制到ARGB格式的Bitmap，得到的只是灰度图，且不符合格式要求，本身就是错误的转换方式。

修复方案 #

方案1：移除错误的Buffer容量检查，正确实现YUV转Bitmap

修改toBitmap方法，跳过错误的容量检查，并使用正确的YUV转RGB逻辑：

private Bitmap toBitmap(Image image) {
    if (image.getFormat() != ImageFormat.YUV_420_888) {
        Log.e("CameraXApp", "Unsupported image format");
        return null;
    }

    Image.Plane[] planes = image.getPlanes();
    ByteBuffer yBuffer = planes[0].getBuffer();
    ByteBuffer uBuffer = planes[1].getBuffer();
    ByteBuffer vBuffer = planes[2].getBuffer();

    int ySize = yBuffer.remaining();
    int uSize = uBuffer.remaining();
    int vSize = vBuffer.remaining();

    byte[] nv21 = new byte[ySize + uSize + vSize];
    // 复制Y平面数据
    yBuffer.get(nv21, 0, ySize);
    // 复制U、V平面数据，注意YUV_420_888的U/V平面可能有 stride，需要按步长复制
    int uStride = planes[1].getRowStride();
    int vStride = planes[2].getRowStride();
    int uPixelStride = planes[1].getPixelStride();
    int vPixelStride = planes[2].getPixelStride();

    int pos = ySize;
    for (int i = 0; i < image.getHeight() / 2; i++) {
        for (int j = 0; j < image.getWidth() / 2; j++) {
            nv21[pos++] = uBuffer.get(i * uStride + j * uPixelStride);
            nv21[pos++] = vBuffer.get(i * vStride + j * vPixelStride);
        }
    }

    // 将NV21数据转Bitmap
    YuvImage yuvImage = new YuvImage(nv21, ImageFormat.NV21, image.getWidth(), image.getHeight(), null);
    ByteArrayOutputStream out = new ByteArrayOutputStream();
    yuvImage.compressToJpeg(new Rect(0, 0, image.getWidth(), image.getHeight()), 100, out);
    byte[] imageBytes = out.toByteArray();
    return BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.length);
}

方案2：直接用TensorFlow Lite处理YUV图像（更高效）

既然最终要转TensorImage，可以跳过Bitmap转换，直接将YUV数据传入TensorFlow Lite，避免格式转换的性能损耗：

@androidx.camera.core.ExperimentalGetImage
private void takeAndAnalyzeImage() {
    if (imageCapture != null) {
        imageCapture.takePicture(cameraExecutor, new ImageCapture.OnImageCapturedCallback() {
            @Override
            public void onCaptureSuccess(ImageProxy image) {
                super.onCaptureSuccess(image);
                Log.d("CameraXApp", "Image capture success. Processing image...");
                Image mediaImage = image.getImage();
                if (mediaImage != null && mediaImage.getFormat() == ImageFormat.YUV_420_888) {
                    try {
                        // 直接将YUV_420_888转换为TensorImage
                        TensorImage tensorImage = new TensorImage(TensorType.UINT8);
                        tensorImage.load(mediaImage, 0, false); // 0表示旋转角度，根据实际情况调整
                        // 调整尺寸到模型输入大小224x224
                        TensorImage resizedTensor = TensorImage.createFrom(tensorImage, new Size(224, 224));
                        runInference(resizedTensor);
                    } catch (Exception e) {
                        Log.e("CameraXApp", "Error processing YUV image: " + e.getMessage());
                    }
                }
                image.close();
            }

            @Override
            public void onError(ImageCaptureException exception) {
                Log.e("CameraXApp", "Photo capture failed: " + exception.getMessage());
            }
        });
    } else {
        Log.e("CameraXApp", "imageCapture is null. Cannot capture image.");
    }
}

额外优化建议 #

• 确保CameraX的ImageCapture配置正确，若需要JPEG格式，可在初始化时设置：

ImageCapture imageCapture = new ImageCapture.Builder()
    .setCaptureMode(ImageCapture.CAPTURE_MODE_MINIMIZE_LATENCY)
    .setTargetRotation(getWindowManager().getDefaultDisplay().getRotation())
    .setOutputFormat(ImageCapture.OUTPUT_FORMAT_JPEG)
    .build();

这样捕获的Image直接是JPEG格式，转换Bitmap会更简单，但可能增加一点延迟。

内容的提问来源于stack exchange，提问作者Ethan W