Open3D显示摄像头流时内存持续增长问题及实时纹理更新解决方案咨询

问题描述 #

我最近在尝试用Open3D把摄像头画面渲染到一个3D平面上，测试代码放在下面了。但运行脚本3-5分钟后，内存占用会开始不停上涨。如果我注释掉modify_geometry_material那一行，内存就不会增长，但纹理要等我移动视角才会更新。后来我改用self.app.post_to_main_thread(self.vis, lambda: self.vis.modify_geometry_material(...))的方式，纹理能实时更新了，但内存还是会在几分钟后开始持续增加。有没有办法实现实时更新摄像头纹理到3D平面同时避免内存泄漏的方法？

测试代码 #

import open3d as o3d
import numpy as np
import cv2
import open3d.visualization.rendering as rendering
import open3d.visualization.gui as gui
import time
import threading

def create_video_plane(width, height):
    # Vertices for a rectangle in the XY plane, centered at (0,0,0)
    vertices = np.array([
        [-width/2, -height/2, 0],
        [ width/2, -height/2, 0],
        [ width/2, height/2, 0],
        [-width/2, height/2, 0]
    ])
    triangles = np.array([
        [0, 1, 2],
        [2, 3, 0]
    ])
    # UV coordinates for each vertex (must match vertices order)
    uvs = np.array([
        [0, 0],  # vertex 0
        [1, 0],  # vertex 1
        [1, 1],  # vertex 2
        [0, 1],  # vertex 3
    ])
    mesh = o3d.geometry.TriangleMesh()
    mesh.vertices = o3d.utility.Vector3dVector(vertices)
    mesh.triangles = o3d.utility.Vector3iVector(triangles)
    mesh.triangle_uvs = o3d.utility.Vector2dVector([
        [0, 0], [1, 0], [1, 1],  # triangle 0
        [1, 1], [0, 1], [0, 0]    # triangle 1
    ])
    mesh.compute_vertex_normals()
    return mesh

class TestApp:
    def __init__(self):
        self.rgb_frame_width = 3840
        self.rgb_frame_height = 2160
        self.ratio = self.rgb_frame_width / self.rgb_frame_height
        self.window_width = 1280
        self.window_height = int(self.window_width / self.ratio)

        self.app = o3d.visualization.gui.Application.instance
        self.app.initialize()
        self.vis = o3d.visualization.O3DVisualizer("Test")
        self.app.add_window(self.vis)

        self.frame = None
        self.image_texture = o3d.geometry.Image(np.zeros((self.rgb_frame_height, self.rgb_frame_width, 3), dtype=np.uint8))
        self.plane = create_video_plane(self.rgb_frame_width, self.rgb_frame_height)
        self.plane.translate([0, 0, self.rgb_frame_width/100])
        self.image_added = False

        self.cap = cv2.VideoCapture(0)
        self.cap.set(cv2.CAP_PROP_FRAME_WIDTH, self.rgb_frame_width)
        self.cap.set(cv2.CAP_PROP_FRAME_HEIGHT, self.rgb_frame_height)
        self.cap.set(cv2.CAP_PROP_FPS, 30)
        self.cap.set(cv2.CAP_PROP_AUTOFOCUS, 0)
        
        self.is_running = True
        self._thread = threading.Thread(target=self.video_loop, daemon=True)
        self._thread.start()

        video_material = rendering.MaterialRecord()
        video_material.shader = "defaultUnlit"
        video_material.base_color = [1.0, 1.0, 1.0, 1.0]
        video_material.albedo_img = self.image_texture
        self.geometries = []
        self.geometries.append({"name":"video_plane", "geometry":self.plane, "material":video_material})

        threading.Thread(target=self.update_thread, daemon=True).start()
        self.app.run()

    def video_loop(self):
        while self.is_running:
            ret, frame = self.cap.read()
            if ret:
                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
                frame = np.ascontiguousarray(frame)
                self.frame = frame  # Store only the raw frame
            time.sleep(1.0 / 30)

    def update_thread(self):
        while(self.frame is None):
            time.sleep(0.1)
        while(self.frame is not None):
            self.geometries[0]['material'].albedo_img = o3d.geometry.Image(self.frame)
            if not self.image_added:
                self.vis.add_geometry(self.geometries[0]['name'], self.geometries[0]['geometry'], self.geometries[0]['material'])
                self.image_added = True
            else:
                self.app.post_to_main_thread(self.vis, lambda: self.vis.modify_geometry_material(self.geometries[0]['name'], self.geometries[0]['material']))
                #self.vis.modify_geometry_material(self.geometries[0]['name'], self.geometries[0]['material'])
            time.sleep(1/20)

app = TestApp()

可能的原因与解决思路 #

我琢磨着内存持续上涨的核心问题应该是每次更新纹理都创建了新的o3d.geometry.Image实例，但旧的纹理资源没被正确回收。Open3D的渲染后端可能没及时清理这些废弃的纹理内存，攒着攒着就导致内存爆涨了。

给你两个实用的优化方向：

1. 复用纹理对象，别每次都新建

不要每次更新都创建新的o3d.geometry.Image，而是初始化时就建好一个固定的纹理对象，之后直接修改它的内部数据就行。这样能避免不断创建新对象带来的内存泄漏：

修改你的update_thread方法：

def update_thread(self):
    while(self.frame is None):
        time.sleep(0.1)
    # 提前拿到纹理的numpy数组引用，避免重复转换
    texture_array = np.asarray(self.image_texture)
    while(self.frame is not None):
        # 直接把新帧的数据复制到已有的纹理数组里，复用同一个纹理对象
        texture_array[:] = self.frame
        if not self.image_added:
            self.vis.add_geometry(self.geometries[0]['name'], self.geometries[0]['geometry'], self.geometries[0]['material'])
            self.image_added = True
        else:
            # 只需要通知渲染器几何数据更新了，不用重新传递整个材质
            self.app.post_to_main_thread(self.vis, lambda: self.vis.update_geometry(self.geometries[0]['name']))
        time.sleep(1/30)  # 和摄像头FPS保持一致，减少不必要的循环

同时，初始化材质的时候，已经把self.image_texture赋值给了video_material.albedo_img，所以后续不需要再修改albedo_img，直接更新纹理对象的内容就行。

2. 清理资源，避免残留

程序退出时记得手动释放摄像头资源，加个销毁方法：

def __del__(self):
    self.is_running = False
    self.cap.release()

这样能确保摄像头设备被正确释放，避免潜在的资源泄漏。

效果验证 #

按上面的修改后，你再跑脚本试试：内存应该不会再持续上涨了，而且摄像头纹理会实时更新到3D平面上，不用移动视角触发刷新。

内容来源于stack exchange