Hololens 2 3D转2D点投影问题：移动设备后结果偏差求助

我通过PhotoCapture获取PhotoCaptureFrame对象并提取外参矩阵，当HoloLens完全静止时投影结果正常，但旋转或平移设备后，3D点到2D图像的投影无法得到理想结果。有没有人使用过HoloLens 2实现3D到2D点投影？

我使用的Unity版本为2022.1.17f1，获取内参和外参矩阵的代码如下：

void OnCapturedPhotoToMemory(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
{
    if (result.success)
    {
        Debug.Log("Saved Photo to disk!");

        if (photoCaptureFrame.TryGetProjectionMatrix(out Matrix4x4 projectionMatrix))
        {
            StreamWriter sw = new StreamWriter(
                Application.persistentDataPath +
                string.Format("/ProjectionMatrix{0}.txt", count)
            );
            
            sw.WriteLine(projectionMatrix.ToString());
            sw.Close();
        }
        else
        {
            Debug.Log("Failed to save camera matrix");
        }

        if (photoCaptureFrame.TryGetCameraToWorldMatrix(out Matrix4x4 cameraMatrix))
        {
            StreamWriter sw = new StreamWriter(
                Application.persistentDataPath + 
                string.Format("/WorldMatrix{0}.txt", count)
            );
            
            sw.WriteLine(cameraMatrix.inverse.ToString());
            sw.Close();
        }
        else
        {
            Debug.Log("Failed to save world matrix");
        }

        StreamWriter s = new StreamWriter(
            Application.persistentDataPath +
            string.Format("/worldToCameraMatrix{0}.txt", count++)
        );
        
        s.WriteLine(cam.worldToCameraMatrix.inverse.ToString());
        s.Close();

        photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);
    }
    else
    {
        Debug.Log("Failed to save Photo to disk");
    }
}

Unity中生成的红色3D点的世界坐标已保存到CSV文件。

我的核心目标是利用内参和外参矩阵将3D点投影到2D图像，使用的Python代码如下：

# readMatrix() 用于读取Unity中PhotoCaptureFrame获取的Matrix4x4矩阵

extrinsic_matrix = readMatrix(f"{WorldMatrices[image_index]}")

# 提取旋转矩阵
rotation_matrix = np.array([row[0:-1] for row in extrinsic_matrix[0:-1]]).copy()
rotation_matrix_ = rotation_matrix.copy()

print("Unity Matrix4x4: ")
print(rotation_matrix_)

# 从OpenGL坐标系转换到OpenCV坐标系
rotation_matrix_[1][0] *= -1
rotation_matrix_[1][2] *= -1
rotation_matrix_[1][2] *= -1
rotation_matrix_[2][0] *= -1
rotation_matrix_[2][3] *= -1
rotation_matrix_[2][2] *= -1

print("Rotation Matrix: ")
print(rotation_matrix_)

# 提取平移向量
translation_vector = np.array([row[-1] for row in extrinsic_matrix[0:-1]]).copy()
translation_vector[1] *= -1
translation_vector[2] *= -1

print("Translation Vector:")
print(translation_vector)

# 读取3D红点坐标，使用cv2.projectPoints进行投影
for key in vertices.keys():
    points, _ = cv2.projectPoints(
        np.float32(vertices[key]), 
        rotation_matrix_, 
        translation_vector, 
        camera_matrix_, 
        None)
    
    for point in points:
        x, y = (point[0][0], point[0][4])
        x = int(x * width/2 +  width/2)
        y = int(y * height/2 +  height/2)
        cv2.circle(image, (x, y), radius=20, color=(255, 0, 0), thickness=-1)

plt.imshow(image[...,::-1])
plt.show()

上述代码中对旋转矩阵和平移向量的部分元素取负，是为了完成从OpenGL到OpenCV坐标系的转换。当前的问题是：投影得到的点未与Unity中生成的红色点重合，这就是我所说的“无法得到理想结果”。

内容的提问来源于stack exchange，提问作者Gigi-G