获取VisionOS中RealityKit立体渲染的单眼视图变换

RealityKit采用单通道立体渲染模式，但仍可获取左右眼各自的视图变换与投影矩阵，核心依赖ARKit提供的相机帧数据接口，以下是具体实现方式：

一、通过ARView获取双眼相机数据 #

直接从ARView的会话帧中提取左右眼的变换矩阵，适用于基于UIKit的AR场景：

import ARKit
import RealityKit

// 初始化ARView
let arView = ARView(frame: .zero)

// 检查当前帧是否可用
if let currentFrame = arView.session.currentFrame {
    let viewportSize = arView.bounds.size
    let interfaceOrientation = UIInterfaceOrientation.portrait
    
    // 获取左眼的视图变换与投影矩阵
    let leftEyeViewTransform = currentFrame.camera.displayTransform(for: interfaceOrientation, 
                                                                     viewportSize: viewportSize, 
                                                                     eye: .left)
    let leftEyeProjectionMatrix = currentFrame.camera.projectionMatrix(for: interfaceOrientation, 
                                                                       viewportSize: viewportSize, 
                                                                       eye: .left)
    
    // 获取右眼的视图变换与投影矩阵
    let rightEyeViewTransform = currentFrame.camera.displayTransform(for: interfaceOrientation, 
                                                                      viewportSize: viewportSize, 
                                                                      eye: .right)
    let rightEyeProjectionMatrix = currentFrame.camera.projectionMatrix(for: interfaceOrientation, 
                                                                        viewportSize: viewportSize, 
                                                                        eye: .right)
    
    // 将矩阵传入自定义渲染逻辑
    // drawSceneWithCameraData(leftEyeViewTransform, leftEyeProjectionMatrix)
    // drawSceneWithCameraData(rightEyeViewTransform, rightEyeProjectionMatrix)
}

二、在SwiftUI RealityView中获取双眼相机数据 #

对于VisionOS常用的SwiftUI环境，可在RealityView的update闭包中访问会话帧数据：

import SwiftUI
import RealityKit
import ARKit

struct StereoCameraView: View {
    var body: some View {
        RealityView { content, attachments in
            // 添加你的RealityKit场景内容
        } update: { content, attachments in
            guard let session = content.session, let currentFrame = session.currentFrame else { return }
            
            let viewportSize = content.viewportSize
            let interfaceOrientation = UIInterfaceOrientation.portrait
            
            // 获取双眼矩阵
            let leftEyeViewTransform = currentFrame.camera.displayTransform(for: interfaceOrientation, 
                                                                             viewportSize: viewportSize, 
                                                                             eye: .left)
            let leftEyeProjection = currentFrame.camera.projectionMatrix(for: interfaceOrientation, 
                                                                         viewportSize: viewportSize, 
                                                                         eye: .left)
            
            let rightEyeViewTransform = currentFrame.camera.displayTransform(for: interfaceOrientation, 
                                                                              viewportSize: viewportSize, 
                                                                              eye: .right)
            let rightEyeProjection = currentFrame.camera.projectionMatrix(for: interfaceOrientation, 
                                                                          viewportSize: viewportSize, 
                                                                          eye: .right)
            
            // 执行自定义渲染操作
        }
    }
}

关键说明 #

• displayTransform对应WebXR中的view.transform.inverse.matrix，是将3D相机空间转换到屏幕空间的视图变换矩阵。
• projectionMatrix直接对应WebXR中的projectionMatrix，用于将相机空间的点投影到裁剪空间。
• 需传入匹配当前界面的方向（如.portrait）和准确的视口尺寸，否则矩阵计算会出现偏差。
• 对于非AR的纯RealityKit场景，系统会自动基于用户的瞳距（IPD）调整双眼矩阵，优先通过ARSession帧获取数据，无需手动计算X轴偏移。

内容的提问来源于stack exchange，提问作者wcochran