ARCore如何提取含相机纹理与OpenGL绘制内容的叠加图像？

Hey there! Sounds like you’ve put in serious work with ARCore—nice job digging through all the docs and examples. The good news is: yes, capturing that full overlay frame (camera feed + your OpenGL-rendered robot) is absolutely possible—you just need to work around the limitations of the Preview 2 TextureReader class.

Here’s why the TextureReader wasn’t cutting it: it’s designed specifically to grab the raw camera YUV texture before any OpenGL overlays are drawn. To get the final composite frame, you need to tap into the OpenGL rendering pipeline itself.

How to Pull Off the Capture #

The standard approach here is to use an OpenGL Framebuffer Object (FBO) to render your entire scene first, then extract the pixel data from that FBO. Here’s a step-by-step breakdown:

1. Set Up a Framebuffer

   • Create an FBO and attach a texture (matching the ARCore camera preview resolution) as its color attachment. This texture will act as the "canvas" for your combined camera feed + OpenGL draws.
   • Example initialization snippet:

     // Create FBO
     int[] fboId = new int[1];
     glGenFramebuffers(1, fboId, 0);
     glBindFramebuffer(GL_FRAMEBUFFER, fboId[0]);
     
     // Create texture for color attachment
     int[] captureTextureId = new int[1];
     glGenTextures(1, captureTextureId, 0);
     glBindTexture(GL_TEXTURE_2D, captureTextureId[0]);
     glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, cameraWidth, cameraHeight, 0, GL_RGBA, GL_UNSIGNED_BYTE, null);
     glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, captureTextureId[0], 0);
     
     // Verify FBO is complete
     if (glCheckFramebufferStatus(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE) {
         // Handle error state
     }
     glBindFramebuffer(GL_FRAMEBUFFER, 0);
     

2. Render to the FBO First

   • In your rendering loop, bind your custom FBO before drawing anything.
   • Draw the ARCore camera texture as you normally would, then layer your OpenGL robot (and any other overlays) on top.
   • After rendering the full scene to the FBO, switch back to the default framebuffer to display the content to the user (so they still see the live AR view).

3. Extract Pixel Data from the FBO

   • When you want to capture the frame, bind your FBO and use glReadPixels to pull the pixel data from the color attachment.
   • Convert the raw pixel data into a usable format (like a Bitmap) for saving or processing:

     glBindFramebuffer(GL_FRAMEBUFFER, fboId[0]);
     ByteBuffer pixelBuffer = ByteBuffer.allocateDirect(cameraWidth * cameraHeight * 4);
     pixelBuffer.order(ByteOrder.nativeOrder());
     glReadPixels(0, 0, cameraWidth, cameraHeight, GL_RGBA, GL_UNSIGNED_BYTE, pixelBuffer);
     glBindFramebuffer(GL_FRAMEBUFFER, 0);
     
     // Convert ByteBuffer to Bitmap
     Bitmap captureBitmap = Bitmap.createBitmap(cameraWidth, cameraHeight, Bitmap.Config.ARGB_8888);
     captureBitmap.copyPixelsFromBuffer(pixelBuffer);
     // Note: You may need to flip the bitmap vertically since OpenGL's origin is bottom-left
     

Key Notes to Avoid Headaches #

• Match Resolution: Ensure your FBO’s texture size exactly matches the ARCore camera preview resolution—otherwise you’ll get stretched or cropped frames.
• GL Thread Safety: All these OpenGL calls must run on the same GL thread that ARCore uses for rendering. Don’t attempt to execute this logic on the main thread.
• Performance: Capturing frames with glReadPixels can be resource-heavy. Trigger captures only when needed (e.g., user taps a button) instead of every frame to avoid framerate drops.

Unfortunately, Preview 2 doesn’t have a built-in API for this composite capture—you have to handle the OpenGL side yourself. Later ARCore versions added more flexible capture options, but for Preview 2, this FBO method is the reliable way to go.

Hope this gets you up and running! Let me know if you hit snags with the implementation.

内容的提问来源于stack exchange，提问作者Tushar Rain