OpenCV立体校正异常：完美合成关键点匹配仍出现Y轴偏移

问题场景 #

我正在使用Blender渲染的合成数据学习立体标定技术，场景可提取像素级完美的关键点对应关系，模拟相机从初始平行状态逐渐严重失准的过程。但使用OpenCV进行图像校正后，单张图像的地平线虽呈水平，但两幅图像间的极线并非水平，即同一场景点的投影连线不水平。已知相机内参，外参由完美关键点对应关系推导，代码如下：

DATA_DIR = "../data/street_sunset_motionblur"
FRAME = 149

# Load images
left_rgb_path  = f"{DATA_DIR}/rgb/left/street_sunset_motionblur_{FRAME:04d}_L.tif"
right_rgb_path = f"{DATA_DIR}/rgb/right/street_sunset_motionblur_{FRAME:04d}_R.tif"
img_left  = cv2.imread(left_rgb_path)
img_right = cv2.imread(right_rgb_path)

# Convert from BGR to RGB
img_left = cv2.cvtColor(img_left, cv2.COLOR_BGR2RGB)
img_right = cv2.cvtColor(img_right, cv2.COLOR_BGR2RGB)

# Load keypoint correspondences.
# Correspondences array has shape (N, 5) where each row is a keypoint and each
# row has the format [x_left, y_left, x_right, y_right, keypoint_id])
keypoints_path = (f"{DATA_DIR}/keypoints/correspondences_{FRAME}.npy")
correspondences = np.load(keypoints_path)

# Split into left points [x_left, y_left] and right points [x_right, y_right] 
points_left  = correspondences[:, :2]
points_right = correspondences[:, 2:4]

# Load intrinsics matrices
K_left  = np.load(f"{DATA_DIR}/camera_matrices/cam_intrinsic_matrix_{FRAME}_L.npy")
K_right = np.load(f"{DATA_DIR}/camera_matrices/cam_intrinsic_matrix_{FRAME}_R.npy")

# Step 1: Compute the Fundamental Matrix
F, mask = cv2.findFundamentalMat(points_left, points_right, cv2.FM_LMEDS)

# Step 2: Compute the Essential Matrix
E = K_right.T @ F @ K_left

# Step 3: Decompose the Essential Matrix to obtain the rotation and translation
_, R, t, _ = cv2.recoverPose(E, points_left, points_right, K_left)

# Specify image size in (width, height) format
image_size = (img_left.shape[1], img_left.shape[0])

# Step 4: Compute the rectification transforms
R1, R2, P1, P2, _, _, _ = cv2.stereoRectify(
    cameraMatrix1=K_left,
    distCoeffs1=None,
    cameraMatrix2=K_right,
    distCoeffs2=None,
    imageSize=image_size,
    R=R,
    T=t,
    flags=cv2.CALIB_ZERO_DISPARITY,
    alpha=-1,
)

# Step 5: Compute the rectification maps
map1_x, map1_y = cv2.initUndistortRectifyMap(K_left, None, R1, P1, image_size, cv2.CV_32FC1)
map2_x, map2_y = cv2.initUndistortRectifyMap(K_right, None, R2, P2, image_size, cv2.CV_32FC1)

# Step 6: Apply the rectification maps
rectified_left  = cv2.remap(img_left, map1_x, map1_y, cv2.INTER_LINEAR)
rectified_right = cv2.remap(img_right, map2_x, map2_y, cv2.INTER_LINEAR)

错误原因分析 #

• 基础矩阵转本质矩阵的冗余误差：已知内参的情况下，无需先计算基础矩阵F再推导本质矩阵E。F针对像素坐标约束，转换过程会引入不必要的计算误差，且cv2.FM_LMEDS鲁棒性算法即使面对完美匹配也会产生微小偏差，不如直接用内参计算E准确。
• recoverPose的解歧义问题：本质矩阵E分解会得到4种旋转平移组合，cv2.recoverPose仅选择保证点在相机前方的解，但该解不一定与Blender场景中相机的真实相对位姿匹配，导致校正变换偏离预期。
• 坐标原点一致性问题：Blender渲染图像默认原点为左下角，而OpenCV像素坐标系原点为左上角，若关键点y坐标未做翻转，会导致垂直方向错位，直接影响外参计算准确性。
• 立体校正参数设置：alpha=-1自动裁剪黑边可能掩盖校正问题；CALIB_ZERO_DISPARITY在相机大角度旋转时，无法保证极线严格水平对齐。

修正方案 #

1. 直接计算本质矩阵替代F转E流程 #

替换原Step1-Step3代码，利用已知内参直接计算E并分解：

# 直接从内参和完美匹配点计算本质矩阵
E, mask = cv2.findEssentialMat(
    points_left, points_right, K_left,
    method=cv2.RANSAC, prob=0.999, threshold=1.0
)
# 分解E得到相对外参，同时用mask过滤无效点
_, R, t, mask = cv2.recoverPose(E, points_left, points_right, K_left, mask=mask)

2. 修正关键点坐标原点 #

若Blender导出的关键点y坐标与OpenCV不一致，执行翻转：

img_height = img_left.shape[0]
# 转换y坐标从Blender坐标系到OpenCV坐标系
points_left[:, 1] = img_height - points_left[:, 1]
points_right[:, 1] = img_height - points_right[:, 1]

3. 调整立体校正参数 #

先保留完整图像观察校正效果，再按需裁剪：

R1, R2, P1, P2, Q, roi_left, roi_right = cv2.stereoRectify(
    cameraMatrix1=K_left,
    distCoeffs1=None,
    cameraMatrix2=K_right,
    distCoeffs2=None,
    imageSize=image_size,
    R=R,
    T=t,
    flags=cv2.CALIB_ZERO_DISPARITY,
    alpha=0,  # 保留所有像素，不自动裁剪黑边，方便检查极线
)
# 后续可根据roi手动裁剪黑边
rectified_left = rectified_left[roi_left[1]:roi_left[1]+roi_left[3], roi_left[0]:roi_left[0]+roi_left[2]]
rectified_right = rectified_right[roi_right[1]:roi_right[1]+roi_right[3], roi_right[0]:roi_right[0]+roi_right[2]]

4. 用Blender真实外参验证（可选） #

若能从Blender导出相机真实相对外参（R_true和t_true），直接替换代码中计算得到的R和t，验证校正效果是否符合预期，排除外参计算问题。

内容的提问来源于stack exchange，提问作者Mandelmus100