如何为WebRTC发送和接收的视频轨道添加模糊及背景模糊效果

需求说明 #

我正在基于WebRTC开发一款视频通话应用，出于安全考虑需要实现两个核心功能：

• 为本地采集的视频添加背景模糊效果
• 对远端陌生用户发起通话时传来的视频默认进行全帧模糊处理，且该模糊效果支持手动关闭

现有代码片段 #

本地视频采集代码 #

private var videoCapturer: CameraVideoCapturer = getCamera()

private fun getCamera(isFrontFacing: Boolean = true): CameraVideoCapturer {
    return Camera1Enumerator(true).run {
        deviceNames.find {
            if (isFrontFacing)
                isFrontFacing(it)
            else
                isBackFacing(it)
        }?.let {
            createCapturer(it, null)
        } ?: throw IllegalStateException()
    }
}


fun startLocalVideoCapture(localVideoOutput: SurfaceViewRenderer, localVideoOutputPiP: SurfaceViewRenderer? = null, localVideoOutputInAppPip: SurfaceViewRenderer? = null, isMicOn: Boolean = true) {
    localVideoOutput.setMirror(true)
    val localVideoSource = peerConnectionFactory.createVideoSource(false)
    val surfaceTextureHelper = SurfaceTextureHelper.create(Thread.currentThread().name, rootEglBase.eglBaseContext)
    (videoCapturer as VideoCapturer).initialize(surfaceTextureHelper, localVideoOutput.context, localVideoSource.capturerObserver)
    videoCapturer.startCapture(1280, 962, 24)
    localVideoTrack = peerConnectionFactory.createVideoTrack(LOCAL_TRACK_ID, localVideoSource)
    val localAudioTrack = peerConnectionFactory.createAudioTrack(
        LOCAL_AUDIO_TRACK_ID, peerConnectionFactory.createAudioSource(MediaConstraints())
    )
    localAudioTrack.setEnabled(isMicOn)
    localVideoTrack?.addSink(localVideoOutput)

    localStream = peerConnectionFactory.createLocalMediaStream(LOCAL_STREAM_ID)
    localStream.audioTracks.add(localAudioTrack)
    localStream.videoTracks.add(localVideoTrack)
    videoSender = peerConnection?.addTrack(localVideoTrack, arrayListOf(LOCAL_STREAM_ID))
    peerConnection?.addTrack(localAudioTrack, arrayListOf(LOCAL_STREAM_ID))
}

远端视频接收代码 #

override fun onAddStream(p0: MediaStream?) {
    super.onAddStream(p0)
    if (p0?.videoTracks?.isNotEmpty() == true) {
        p0.videoTracks?.get(0)?.addSink(remote_view)
        remoteVideoTrack = p0.videoTracks?.get(0)
        callControlsViewModel.isClientCamOn.postValue(true)
    }
    if (p0?.audioTracks?.isNotEmpty() == true) {
        remoteAudioTrack = p0.audioTracks?.get(0)
        callControlsViewModel.isClientMicOn.postValue(true)
    }
}

实现方案 #

一、本地视频背景模糊实现 #

WebRTC原生没有内置背景模糊能力，需要通过视频帧预处理实现，推荐两种方案：

方案1：基于GPUImage的模糊滤镜

利用GPUImage库对采集到的视频帧进行高斯模糊，若需精准保留人像，可结合人像分割能力。

修改本地采集流程：

1. 引入GPUImage依赖：implementation 'jp.co.cyberagent.android:gpuimage:2.1.0'
2. 包装VideoCapturer.CapturerObserver，在帧捕获回调中完成模糊处理后再传递给原Observer

核心修改示例：

// 自定义CapturerObserver包装类，处理帧模糊
class BlurCapturerObserver(private val originalObserver: VideoCapturer.CapturerObserver, private val context: Context) : VideoCapturer.CapturerObserver {
    private val gpuImage = GPUImage(context)
    private val blurFilter = GPUImageGaussianBlurFilter(25f) // 模糊程度，数值越大效果越强

    override fun onCapturerStarted(success: Boolean) {
        originalObserver.onCapturerStarted(success)
    }

    override fun onCapturerStopped() {
        originalObserver.onCapturerStopped()
    }

    override fun onFrameCaptured(frame: VideoFrame) {
        // 将VideoFrame转为Bitmap处理
        val bitmap = frame.toBitmap()
        gpuImage.setImage(bitmap)
        val blurredBitmap = gpuImage.getBitmapWithFilterApplied(blurFilter)
        // 转换回VideoFrame并传递给原Observer
        val newFrame = VideoFrame(blurredBitmap, frame.rotation, frame.timestampNs)
        originalObserver.onFrameCaptured(newFrame)
        // 释放资源
        bitmap.recycle()
        blurredBitmap.recycle()
        newFrame.release()
    }
}

// 在startLocalVideoCapture中替换原CapturerObserver
(videoCapturer as VideoCapturer).initialize(
    surfaceTextureHelper,
    localVideoOutput.context,
    BlurCapturerObserver(localVideoSource.capturerObserver, localVideoOutput.context)
)

方案2：基于ML Kit人像分割的精准背景模糊

若需仅模糊背景、保留清晰人像，使用Google ML Kit人像分割能力：

1. 引入ML Kit依赖：implementation 'com.google.mlkit:segmentation-selfie:16.0.0-beta5'
2. 对采集帧进行人像分割，提取掩码后仅模糊背景区域

二、远端视频全模糊实现 #

远端视频模糊通过拦截VideoTrack渲染流程实现，自定义Sink处理模糊逻辑，同时支持开关控制：

1. 创建自定义模糊Sink类：

class RemoteBlurSink(private val targetSink: VideoSink, private var isBlurEnabled: Boolean, private val context: Context) : VideoSink {
    private val gpuImage = GPUImage(context)
    private val blurFilter = GPUImageGaussianBlurFilter(30f)

    // 提供开关控制方法
    fun setBlurEnabled(enabled: Boolean) {
        isBlurEnabled = enabled
    }

    override fun onFrame(frame: VideoFrame) {
        if (!isBlurEnabled) {
            targetSink.onFrame(frame)
            return
        }
        // 对帧进行全模糊处理
        val bitmap = frame.toBitmap()
        gpuImage.setImage(bitmap)
        val blurredBitmap = gpuImage.getBitmapWithFilterApplied(blurFilter)
        val newFrame = VideoFrame(blurredBitmap, frame.rotation, frame.timestampNs)
        targetSink.onFrame(newFrame)
        // 释放资源
        bitmap.recycle()
        blurredBitmap.recycle()
        newFrame.release()
    }
}

2. 修改远端视频接收代码，替换原Sink并添加开关逻辑：

private var remoteBlurSink: RemoteBlurSink? = null

override fun onAddStream(p0: MediaStream?) {
    super.onAddStream(p0)
    if (p0?.videoTracks?.isNotEmpty() == true) {
        val remoteVideoTrack = p0.videoTracks[0]
        this.remoteVideoTrack = remoteVideoTrack
        callControlsViewModel.isClientCamOn.postValue(true)
        
        // 初始化模糊Sink，陌生用户场景默认开启模糊
        remoteBlurSink = RemoteBlurSink(remote_view, true, remote_view.context)
        remoteVideoTrack.addSink(remoteBlurSink)
        remoteVideoTrack.removeSink(remote_view)
    }
    if (p0?.audioTracks?.isNotEmpty() == true) {
        remoteAudioTrack = p0.audioTracks[0]
        callControlsViewModel.isClientMicOn.postValue(true)
    }
}

// 外部调用的开关控制方法
fun toggleRemoteBlur(enabled: Boolean) {
    remoteBlurSink?.setBlurEnabled(enabled)
}

注意事项 #

• 视频帧处理会消耗额外性能，建议在高配置设备启用，或提供性能档位选择
• 避免在主线程处理帧转换和模糊，可通过SurfaceTextureHelper指定后台线程
• ML Kit人像分割依赖设备NNAPI/GPU加速，CPU处理模式性能会大幅下降

内容的提问来源于stack exchange，提问作者Ivan Šimović