如何基于Camera2 API在录制视频时添加自定义滤镜

Hey there! Great question—custom filters with Camera2 do require a bit more work than using the built-in CONTROL_EFFECT_MODE parameters, but it's totally achievable for real-time video. Let me break down the most practical approaches you can use:

This is the go-to method for smooth, hardware-accelerated custom filters (just like apps like Retrica use). It works by routing Camera2's video frames through an OpenGL pipeline where you apply custom shader effects.

Step-by-Step Overview: #

• Route Camera2 output to a TextureView: Configure your CaptureSession to send frames to a TextureView—this gives you access to the frame as an OpenGL texture.
• Set up an OpenGL ES context: Create a GLSurfaceView or use a custom renderer to handle shader compilation and frame rendering.
• Write custom fragment shaders: Shaders are small programs that run on the GPU to manipulate pixel colors. You can define any effect here (e.g., vintage tones, color grading, edge detection).
• Render processed frames: In the TextureView's onSurfaceTextureUpdated callback, grab the current frame texture, pass it through your shader, and render the result to both the preview and your MediaRecorder's surface (for recording).

Example Custom Warm-Tone Shader: #

// Fragment Shader (warm_filter.frag)
precision mediump float;
varying vec2 v_TexCoord;
uniform sampler2D u_Texture;

void main() {
    vec4 originalColor = texture2D(u_Texture, v_TexCoord);
    
    // Boost red/green channels, dim blue to create warm tones
    originalColor.r = min(originalColor.r * 1.25, 1.0);
    originalColor.g = min(originalColor.g * 1.15, 1.0);
    originalColor.b = max(originalColor.b * 0.8, 0.0);
    
    gl_FragColor = originalColor;
}

If you prefer to avoid OpenGL and want to work directly with pixel data (for simpler filters), use ImageReader to capture frames and process them on the CPU. Note: This is less performant for high-resolution video, but works well for lightweight effects.

Step-by-Step Overview: #

• Create an ImageReader: Initialize it with ImageFormat.YUV_420_888 (Camera2's default output format) and match your camera's resolution.
• Add ImageReader's Surface to CaptureSession: Include it alongside your preview and recording surfaces.
• Process frames in the callback: Use setOnImageAvailableListener to grab each frame, convert the YUV data to a bitmap or raw pixel array, apply your filter logic, then pass the processed frame to your recorder or preview.

Example Brightness Adjustment (CPU Processing): #

imageReader.setOnImageAvailableListener(reader -> {
    try (Image image = reader.acquireNextImage()) {
        Image.Plane yPlane = image.getPlanes()[0];
        ByteBuffer yBuffer = yPlane.getBuffer();
        byte[] yData = new byte[yBuffer.remaining()];
        yBuffer.get(yData);

        // Increase brightness (Y channel controls luminance, range: 16-235)
        for (int i = 0; i < yData.length; i++) {
            int luminance = yData[i] & 0xFF;
            luminance = Math.min(luminance + 30, 235);
            yData[i] = (byte) luminance;
        }

        // Write modified data back and route to recording/preview
        yBuffer.position(0);
        yBuffer.put(yData);
        // ... pass to MediaRecorder or convert to Bitmap for preview
    }
}, backgroundHandler);

RenderScript is a middle ground between OpenGL and raw CPU processing—it uses hardware acceleration but lets you write simpler, C-like code for filters. It's great if you don't want to dive into shader syntax.

Step-by-Step Overview: #

• Create a RenderScript context: Initialize it with your app's context.
• Write a RenderScript kernel: Define your filter logic in a .rs file (e.g., blur, color inversion).
• Process frames: Convert Image data from ImageReader into Allocation objects, run the kernel, then output the processed data to your recording/preview surface.

Example Blur Filter (RenderScript): #

// blur_filter.rs
#pragma version(1)
#pragma rs java_package_name(com.your.app.package)

rs_allocation input;
rs_allocation output;

void __attribute__((kernel)) blur(uint32_t x, uint32_t y) {
    // 3x3 box blur
    float4 sum = 0.0f;
    for (int dx = -1; dx <= 1; dx++) {
        for (int dy = -1; dy <= 1; dy++) {
            sum += rsGetElementAt_float4(input, x + dx, y + dy);
        }
    }
    rsSetElementAt_float4(output, sum / 9.0f, x, y);
}

Key Tips for Success: #

• Prioritize hardware acceleration: OpenGL or RenderScript will give you smooth performance for 1080p/4K video—CPU processing may lag on lower-end devices.
• Sync preview and recording: Make sure your processed frames are sent to both the preview surface and MediaRecorder's surface to avoid desync.
• Handle color spaces: Camera2 outputs YUV data by default—convert to RGB if needed, but try to process in YUV where possible for better performance.

内容的提问来源于stack exchange，提问作者Usama Shakeel