Android WebRTC中OpenGL转换接收端VideoFrame为Bitmap时崩溃

现象 #

• 应用处理WebRTC收发流，需将VideoFrame转换为Bitmap以录制接收流，通过VideoSink处理帧回调
• 传出VideoFrame转Bitmap完全正常，但传入VideoFrame转换时直接触发OpenGL引擎崩溃

报错信息 #

首次崩溃报错：

java.lang.IllegalStateException: Framebuffer not complete, status: 0

后续每帧重复报错：

java.lang.RuntimeException: glCreateShader() failed. GLES20 error: 0
java.lang.NullPointerException: Attempt to invoke virtual method 'void org.webrtc.GlShader.useProgram()' on a null object reference

崩溃触发点 #

崩溃发生在以下代码行：

bitmapTextureFramebuffer.setSize(scaledWidth, scaledHeight); 

相关代码 #

主服务帧转换逻辑 #

private VideoFrameDrawer frameDrawer;
private GlRectDrawer drawer;
boolean mirrorHorizontally = true;
boolean mirrorVertically = true;
private final Matrix drawMatrix = new Matrix();
private GlTextureFrameBuffer bitmapTextureFramebuffer ;
private boolean isInitGPU = false;

public Bitmap GetBitmapFromVideoFrame(VideoFrame frame){
    try {

        if (!isInitGPU)
        {
            bitmapTextureFramebuffer =
                    new GlTextureFrameBuffer(GLES20.GL_RGBA);
            isInitGPU = true;
            frameDrawer = new VideoFrameDrawer();
            drawer = new GlRectDrawer();
        }
        drawMatrix.reset();
        drawMatrix.preTranslate(0.5f, 0.5f);
        drawMatrix.preScale(mirrorHorizontally ? -1f : 1f, mirrorVertically ? -1f : 1f);
        drawMatrix.preScale(1f, -1f);//We want the output to be upside down for Bitmap.
        drawMatrix.preTranslate(-0.5f, -0.5f);

        final int scaledWidth = (int) (1 * frame.getRotatedWidth());
        final int scaledHeight = (int) (1 * frame.getRotatedHeight());

        bitmapTextureFramebuffer.setSize(scaledWidth, scaledHeight);

        GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, bitmapTextureFramebuffer.getFrameBufferId());
        GLES20.glFramebufferTexture2D(GLES20.GL_FRAMEBUFFER, GLES20.GL_COLOR_ATTACHMENT0,
                GLES20.GL_TEXTURE_2D, bitmapTextureFramebuffer.getTextureId(), 0);

        GLES20.glClearColor(0/* red */, 0/* green */, 0/* blue */, 0/* alpha */);
        GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);
        frameDrawer.drawFrame(frame, drawer, drawMatrix, 0/* viewportX */,
                0/* viewportY */, scaledWidth, scaledHeight);

        final ByteBuffer bitmapBuffer = ByteBuffer.allocateDirect(scaledWidth * scaledHeight * 4);
        GLES20.glViewport(0, 0, scaledWidth, scaledHeight);
        GLES20.glReadPixels(
                0, 0, scaledWidth, scaledHeight, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, bitmapBuffer);

        GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, 0);
        GlUtil.checkNoGLES2Error("EglRenderer.notifyCallbacks");

        final Bitmap bitmap = Bitmap.createBitmap(scaledWidth, scaledHeight, Bitmap.Config.ARGB_8888);
        bitmap.copyPixelsFromBuffer(bitmapBuffer);
        return bitmap;

     } catch (Exception e) {
        Log.e(TAG, e.toString());
        return null;
    }
}

VideoSink实现 #

private class ProxyVideoSink implements VideoSink {
  private VideoSink target;
  @Override
  synchronized public void onFrame(VideoFrame frame) {
      //send the frame to the service to handle
      if (onStreamWebRTCListener != null )
           onStreamWebRTCListener.OnStreamBitmapListener(frame);
      target.onFrame(frame);
  }
}

排查细节 #

• 首次崩溃时bitmapTextureFramebuffer参数：width=640，height=480，frameBufferId=1，textureId=2，format=6408（GL_RGBA）
• 传入VideoFrame运行在"IncomingVideoSt - 23064"专用线程，分辨率640x480；传出VideoFrame运行在main线程，分辨率360x540
• 尝试过用runOnUiThread切换到主线程执行转换，无效

解决方案建议 #

核心问题：OpenGL上下文线程绑定 #

WebRTC传入帧运行在专用视频线程，而OpenGL资源（GlTextureFrameBuffer、GlRectDrawer等）的创建和操作必须在同一个拥有有效EGL上下文的线程中执行，跨线程操作会导致上下文丢失、资源创建失败。

1. 创建专用OpenGL处理线程
   • 初始化带EGL上下文的后台线程（比如用HandlerThread），所有帧转换任务都投递到该线程执行，确保OpenGL资源的初始化、操作全在同一上下文内。
2. 修复线程安全问题
   • isInitGPU的判断和资源初始化是非线程安全的，需加锁确保只初始化一次，且在目标线程内完成初始化。
3. 添加FrameBuffer状态检查
   • 在bitmapTextureFramebuffer.setSize后增加状态验证：

     int status = GLES20.glCheckFramebufferStatus(GLES20.GL_FRAMEBUFFER);
     if (status != GLES20.GL_FRAMEBUFFER_COMPLETE) {
         throw new IllegalStateException("Framebuffer not complete, status: " + status);
     }
     

   • 部分老旧设备对纹理分辨率有2的幂次限制，可尝试将帧分辨率调整为符合要求的值。

替代转换方案 #

1. 参考WebRTC内置VideoFileRenderer实现 #

WebRTC的VideoFileRenderer已经封装了帧渲染转Bitmap的逻辑，可直接参考其EglRenderer的帧处理流程，避免重复造轮子。

2. RenderScript加速YUV转Bitmap #

跳过OpenGL，直接用RenderScript将VideoFrame的YUV数据转换为Bitmap，性能更稳定：

// 从VideoFrame获取I420格式数据
VideoFrame.Buffer buffer = frame.getBuffer();
VideoFrame.I420Buffer i420Buffer = buffer.toI420();

// 初始化RenderScript
RenderScript rs = RenderScript.create(context);
ScriptIntrinsicYuvToRGB yuvToRgbIntrinsic = ScriptIntrinsicYuvToRGB.create(rs, Element.U8_4(rs));

// 创建YUV输入分配器
Type.Builder yuvType = new Type.Builder(rs, Element.U8(rs))
        .setX(i420Buffer.getWidth())
        .setY(i420Buffer.getHeight())
        .setYuvFormat(ImageFormat.YUV_420_888);
Allocation inAlloc = Allocation.createTyped(rs, yuvType.create(), Allocation.USAGE_SCRIPT);

// 创建ARGB输出分配器
Type.Builder rgbType = new Type.Builder(rs, Element.RGBA_8888(rs))
        .setX(i420Buffer.getWidth())
        .setY(i420Buffer.getHeight());
Allocation outAlloc = Allocation.createTyped(rs, rgbType.create(), Allocation.USAGE_SCRIPT | Allocation.USAGE_IO_OUTPUT);

// 填充YUV数据
inAlloc.copyFrom(i420Buffer.getData());

// 执行转换
yuvToRgbIntrinsic.setInput(inAlloc);
yuvToRgbIntrinsic.forEach(outAlloc);

// 转换为Bitmap
Bitmap bitmap = Bitmap.createBitmap(i420Buffer.getWidth(), i420Buffer.getHeight(), Bitmap.Config.ARGB_8888);
outAlloc.copyTo(bitmap);

// 释放资源
i420Buffer.release();
rs.destroy();

内容的提问来源于stack exchange，提问作者kfir