如何为Open3D的O3DVisualizer类添加面板或额外窗口（Python）

背景 #

我正尝试为concept-graphs映射系统的映射脚本添加GUI。目前代码较为粗糙，可查看项目内的conceptgraph/slam/gui_realtime_mapping.py文件。

该映射系统以RGB图像、深度图像及位姿为输入，迭代构建包含语义特征的基于对象的3D地图。我希望可视化整个地图构建过程，以便在构建时检查地图并调试问题。

我查看了Python示例，发现multiple_windows.py示例具备我所需的功能，即可可视化对象点云和边界框等，目前这部分运行正常。

问题 #

我还希望显示更多信息：在地图旁的面板中展示当前RGB图像、深度图像、标注图像、当前帧编号、当前对象数量等内容。

我期望实现类似dense_slam_gui.py示例的效果。

我尝试了多种方法，但无法为o3d.visualization.O3DVisualizer添加面板，也无法添加可同步更新的其他窗口。我不介意是否为同一窗口，只要能在地图旁显示这些信息以优化映射系统即可。

我的代码 #

以下是相关代码片段。

应用创建方式（可见我尝试添加其他窗口或面板，但均出现错误）： #

def run(self):
    app = o3d.visualization.gui.Application.instance
    app.initialize()

    self.main_vis = o3d.visualization.O3DVisualizer(
        "Open3D - Multi-Window Demo")
    self.main_vis.add_action("Take snapshot in new window",
                             self.on_snapshot)
    self.main_vis.add_action("Pause/Resume updates", lambda vis: self.toggle_pause())
    self.main_vis.set_on_close(self.on_main_window_closing)
    
    app.add_window(self.main_vis)
    # app.add_window(self.images_window)
    
    # # Setup the secondary window for images
    # # self.setup_image_display_window()
    # # self.image_window = app.create_window("RGB and Depth Images", 640, 480)
    # self.image_window = gui.Application.instance.create_window("RGB and Depth Images", 640, 480)
    # # self.image_window.create_window()
    # # self.image_window = o3d.visualization.Visualizer("RGB and Depth Images", 640, 480)
    # self.layout = o3d.visualization.gui.Vert(0, o3d.visualization.gui.Margins(10))
    # self.image_window.add_child(self.layout)

    # # Create image widgets
    # self.rgb_widget = o3d.visualization.gui.ImageWidget()
    # self.depth_widget = o3d.visualization.gui.ImageWidget()

    # # Add image widgets to the layout
    # self.layout.add_child(self.rgb_widget)
    # self.layout.add_child(self.depth_widget)
    # app.add_window(self.image_window)

    self.snapshot_pos = (self.main_vis.os_frame.x, self.main_vis.os_frame.y)
    threading.Thread(target=self.update_thread).start()
    app.run()

更新线程的实现： #

def update_thread(self):
    # This is NOT the UI thread, need to call post_to_main_thread() to update
    # the scene or any part of the UI.
    
    ... lots of code, basically the whole mapping script ...
    
        def my_update_cloud():                
            
            # Remove previous objects
            for obj_name in self.prev_obj_names:
                self.main_vis.remove_geometry(obj_name)
            self.prev_obj_names = []
            
            # Remove previous bounding boxes
            for bbox_name in self.prev_bbox_names:
                self.main_vis.remove_geometry(bbox_name)
            self.prev_bbox_names = []
            
            # Add the new objects and bounding boxes
            for obj_num, obj in enumerate(self.objects):
                obj_label = f"{obj['curr_obj_num']}_{obj['class_name']}"
                
                obj_name = f"obj_{obj_label}"
                bbox_name = f"bbox_{obj_label}"
                
                self.prev_obj_names.append(obj_name)
                self.main_vis.add_geometry(obj_name, obj['pcd'])
                
                self.prev_bbox_names.append(bbox_name)
                self.main_vis.add_geometry(bbox_name, obj['bbox'] )
                
        if self.is_done:  # might have changed while sleeping
            break
        o3d.visualization.gui.Application.instance.post_to_main_thread(
            self.main_vis, my_update_cloud)

我的需求 #

首先，我希望能更新显示当前RGB、深度图像及帧编号，大致实现如下：

def update_images(self):
    # Convert numpy images to Open3D images and update widgets
    o3d_rgb_image = o3d.geometry.Image(self.curr_image_rgb)
    o3d_depth_image = o3d.geometry.Image((self.curr_depth_array / self.curr_depth_array.max() * 255).astype(np.uint8))  # Example normalization

    self.rgb_widget.update_image(o3d_rgb_image)
    self.depth_widget.update_image(o3d_depth_image)
    
# and then in the main update thread loop I would do        
self.curr_image_rgb = image_rgb
self.curr_depth_array = depth_array
o3d.visualization.gui.Application.instance.post_to_main_thread(
    self.image_window, self.update_images())

希望上述需求清晰明了，若能得到方向指引将不胜感激。我已查阅资料及咨询ChatGPT，但未获有效解决方案。

感谢您的时间与帮助。

内容的提问来源于stack exchange，提问作者Ali Kuwajerwala