如何从MediaProjection屏幕捕获的SurfaceView提取RGB值？优先用RenderScript提速

Hey there! Glad to hear your MediaProjectionDemo is up and running. Extracting RGB values from that SurfaceView holding your screen capture is totally feasible, and leaning on RenderScript is a great call for speed—it taps into hardware acceleration on Android devices, which makes a huge difference for frame processing. Let’s walk through how to implement this step by step.

Core Concept #

First, we need to capture the current frame data from the SurfaceView, then feed that data into a RenderScript kernel to efficiently extract RGB components. Since SurfaceView renders content to a hardware surface, we’ll use the PixelCopy API to copy that surface content into a Bitmap, which we can then pass to RenderScript.

Step 1: Initialize the RenderScript Environment #

Start by setting up the RenderScript context and linking to our custom processing script. If you’re targeting older Android versions, make sure you’ve included the RenderScript support library (modern Android versions have it built-in).

// Initialize RenderScript context
val rs = RenderScript.create(applicationContext)
// Create our custom RenderScript kernel object (we'll define this next)
val rgbExtractorScript = ScriptC_rgb_extractor(rs)

Step 2: Write the RenderScript Kernel #

Create a new file named rgb_extractor.rs in your src/main/rs directory. This kernel will process each pixel in the frame, extract its RGB values, and store them in a flattened output array.

#pragma version(1)
#pragma rs java_package_name(com.your.package.here) // Replace with your app's package name

// Input allocation (expects ARGB_8888 format, standard for Android Bitmaps)
rs_allocation inputFrame;
// Output array to hold flattened RGB values: [r1, g1, b1, r2, g2, b2, ...]
uint32_t* rgbOutput;
// Dimensions of the captured frame
int frameWidth;
int frameHeight;

// Kernel function that runs on every pixel
void root(const uchar4* pixelIn, uint32_t* pixelOut, uint32_t x, uint32_t y) {
    // Extract RGB components (ignore alpha if you don't need it)
    uint8_t red = pixelIn->r;
    uint8_t green = pixelIn->g;
    uint8_t blue = pixelIn->b;
    
    // Calculate index in the flattened output array
    int outputIndex = (y * frameWidth + x) * 3;
    rgbOutput[outputIndex] = red;
    rgbOutput[outputIndex + 1] = green;
    rgbOutput[outputIndex + 2] = blue;
}

Step 3: Capture the SurfaceView Frame and Process It #

Use the PixelCopy API to grab the current frame from the SurfaceView, convert it to a RenderScript allocation, and run our kernel to extract RGB values.

val surfaceView = findViewById<SurfaceView>(R.id.your_surface_view_id) // Replace with your SurfaceView ID

// Create a Bitmap matching the SurfaceView's dimensions
val captureBitmap = Bitmap.createBitmap(
    surfaceView.width,
    surfaceView.height,
    Bitmap.Config.ARGB_8888
)

// Request PixelCopy to capture the current frame
PixelCopy.request(
    surfaceView,
    captureBitmap,
    { copyResult ->
        if (copyResult == PixelCopy.SUCCESS) {
            // Convert the captured Bitmap to a RenderScript allocation
            val inputAllocation = Allocation.createFromBitmap(rs, captureBitmap)
            
            // Calculate the size of our output array (width * height * 3 RGB values)
            val outputSize = captureBitmap.width * captureBitmap.height * 3
            val rgbResults = IntArray(outputSize)
            val outputAllocation = Allocation.createSized(
                rs,
                Element.U32(rs),
                outputSize
            )
            
            // Pass parameters to our RenderScript script
            rgbExtractorScript._inputFrame = inputAllocation
            rgbExtractorScript._rgbOutput = outputAllocation
            rgbExtractorScript._frameWidth = captureBitmap.width
            rgbExtractorScript._frameHeight = captureBitmap.height
            
            // Run the kernel across the entire frame
            rgbExtractorScript.forEach_root(inputAllocation, outputAllocation)
            
            // Copy the processed RGB values from the allocation to our array
            outputAllocation.copyTo(rgbResults)
            
            // Now you can use rgbResults! For example, access pixel (x,y):
            // val pixelIndex = (y * captureBitmap.width + x) * 3
            // val red = rgbResults[pixelIndex]
            // val green = rgbResults[pixelIndex + 1]
            // val blue = rgbResults[pixelIndex + 2]
            
            // Clean up allocations to avoid memory leaks
            inputAllocation.destroy()
            outputAllocation.destroy()
        } else {
            // Handle PixelCopy failure (log or show an error)
            Log.e("RGBExtractor", "Frame capture failed with result: $copyResult")
        }
    },
    Handler(Looper.getMainLooper())
)

Pro Tips for Optimization #

• Reuse Allocations: If your SurfaceView’s size doesn’t change, reuse the same inputAllocation and outputAllocation for every frame instead of creating new ones. This cuts down on memory overhead and garbage collection.
• YUV Format Handling: If your MediaProjection captures frames in YUV format (common for some screen capture setups), use RenderScript’s built-in ScriptIntrinsicYuvToRGB to convert to RGB first, then run our extraction kernel.
• Throttle Frame Processing: You don’t need to process every single frame unless your use case demands it. Add a timer to process frames at a fixed interval (e.g., 10fps) to save battery and CPU.

Fallback Option (No RenderScript) #

If for some reason you can’t use RenderScript, you can directly extract RGB values from the Bitmap—but this is significantly slower for large frames:

val pixelArray = IntArray(captureBitmap.width * captureBitmap.height)
captureBitmap.getPixels(pixelArray, 0, captureBitmap.width, 0, 0, captureBitmap.width, captureBitmap.height)

for (i in pixelArray.indices) {
    val argbPixel = pixelArray[i]
    val red = Color.red(argbPixel)
    val green = Color.green(argbPixel)
    val blue = Color.blue(argbPixel)
    // Store or use the RGB values here
}

内容的提问来源于stack exchange，提问作者CompNeuroDev