三维重建姿态出现旋转偏移的原因排查与解决咨询

我有足球比赛的两个不同视角视频，先在两个视角上用2D姿态估计器提取2D人体姿态点，再手动匹配两个相机下的2D姿态点建立对应关系。目前三维重建出的姿态形态看起来合理，但整体在空间中存在旋转和偏移，找不到原因。

我的三角化姿态结果如图所示。

相机标定实现 #

为标定相机，我手动标记了场地中的已知点（如场地角点、点球点等平面点，也用到球门横梁交点这类非平面点）。尝试多种方法后，最佳结果是通过平面点调用cameraCalibrate函数估计内参矩阵，核心实现函数如下：

def new_guess(self, image_points, real_worlds):
        planar_img, planar_3d = Homo_est.get_planar_points(image_points, real_worlds)
        mat, ds = self.create_camera_guess(planar_img, planar_3d)
        real_worlds = np.array([list(tup) for tup in real_worlds])
        image_points =[list(tup) for tup in image_points]
        objPts = []
        imgPts = []
        
        objPts.append(real_worlds)
        imgPts.append(image_points)
        objPts = np.float32(objPts)
        imgPts = np.float32(imgPts)
        dist_coeffs = np.zeros((4,1)) 
        gray = cv2.cvtColor(self.img, cv2.COLOR_BGR2GRAY)
        #mat = self.second_camera_guess() # bästa so far
        criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30000, 0.0001) # att ha 
        ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objPts, imgPts, gray.shape[::-1], mat, ds, flags=cv2.CALIB_USE_INTRINSIC_GUESS, criteria=criteria)
        return mtx, dist

P矩阵计算（基于solvePNP） #

将上述得到的内参矩阵传入solvePNP函数计算投影矩阵P，并用cv2.projectPoints()验证结果，投影误差最多10像素（多数情况更小），这个精度我认为是合格的。核心实现函数如下：

def calibrate_solvePNP(self, image_points, real_worlds):
        planar_img, planar_3d = Homo_est.get_planar_points(image_points, real_worlds)    
        mat, dist = self.new_guess(image_points, real_worlds)
        real_worlds = np.array([list(tup) for tup in real_worlds])
        image_points =[list(tup) for tup in image_points]
        objPts = []
        imgPts = []        
        objPts.append(real_worlds)
        imgPts.append(image_points)
        objPts = np.float32(objPts)
        imgPts = np.float32(imgPts)
        dist_coeffs = np.zeros((4,1)) 
        gray = cv2.cvtColor(self.img, cv2.COLOR_BGR2GRAY)
        (succsess, rvec, t) = cv2.solvePnP(objPts, imgPts, mat, dist_coeffs)
        res, jac = cv2.projectPoints(real_worlds, rvec, t, mat, dist_coeffs)

        R, jac = cv2.Rodrigues(rvec)
        Rt = np.concatenate([R,t], axis=-1) # [R|t]
        P = np.matmul(mat, Rt)
        return P 

三角化实现 #

完成相机参数计算后，对匹配的姿态点进行三角化，核心实现如下：

def triangulate_pts(self):
        P1_points = Homo_est.player_view1() # pose coordinates view 1
        P2_points = Homo_est.player_view2() # pose coordinates view 2
        homo = Homo_est()
        P1, P2, K1, K2, d1, d2 = homo.generate_cameras() # essentially calls my calibrate_solvePNP and new_guess to create these values
        dist_coeffs = np.zeros((4,1)) 
        P1_new = np.matmul(np.linalg.inv(K1), P1) # unsure if this is good or bad
        P2_new = np.matmul(np.linalg.inv(K2), P2)

        # doesn't seem to make a difference using d1 or d2
        P1_undist = cv2.undistortPoints(P1_points, cameraMatrix=K1, distCoeffs=dist_coeffs)
        
        P2_undist = cv2.undistortPoints(P2_points, 
                                   cameraMatrix=K2,
                                   distCoeffs=dist_coeffs)
        
        triangulation = cv2.triangulatePoints(P1_new, P2_new, P1_undist, P2_undist)

        homog_points = triangulation.transpose()
        
        euclid_points = cv2.convertPointsFromHomogeneous(homog_points)

euclid_points就是我用于绘制三维姿态的点集。

已尝试的解决方案 #

• 调整畸变系数的各种参数，对结果影响不大；
• 尝试将其中一个相机归一化为[I 0]并转换另一个相机的参数，但操作后未解决问题（可能操作有误）；
• 遵循标准三角化流程进行实现。

求助问题 #

我几乎尝试了所有能想到的方法，直觉上认为肯定存在消除这种旋转和偏移的方式——毕竟重建出的姿态形态是正确的。最坏情况下，只要能消除旋转即可（因为我已经有场地平面点的准确单应性）。请问导致这种旋转和偏移的可能原因是什么？

内容的提问来源于stack exchange，提问作者emil holmberg