鱼眼图像去畸变获取俯视视角：如何正确处理相机姿态？

我尝试从鱼眼畸变图像中获取俯视视角。当前相机姿态如下：

• 从左视图看，相机向前轴下方倾斜40deg；
• 从俯视图看，相机向前轴左侧倾斜30deg。

我编写了Python3脚本，但对输出结果不满意，怀疑相机姿态的处理方式存在问题。

import cv2
import numpy as np

rollDeg = 50
pitchDeg = 30
# Camera calibration parameters
K = np.array([[413.4459003089536, 0.0, 625.4657090067509],
          [0.0, 414.5602416017697, 393.99252903112915],
          [0.0, 0.0, 1.0]])

D = np.array([[0.2422953376828772],
          [0.5059306403608751],
          [-0.48731143899233426],
          [0.17754105229960523]])

# Estimate a new camera matrix optimized for undistortion
new_K = cv2.fisheye.estimateNewCameraMatrixForUndistortRectify(K, D, (1280, 720), np.eye(3), balance=1, new_size=(1280, 720), fov_scale=1)

# Initial values for new_K parameters
fx_init = 230
fy_init = 230
cx_init = 1280//2
cy_init = 720//2

# Function to update undistorted image
def update_image():
# Update new_K matrix based on trackbar values
rollDeg    = cv2.getTrackbarPos('roll', 'img')
pitchDeg      = cv2.getTrackbarPos('pitch', 'img')
new_K[0, 0] = cv2.getTrackbarPos('fx', 'img')
new_K[1, 1] = cv2.getTrackbarPos('fy', 'img')
new_K[0, 2] = cv2.getTrackbarPos('cx', 'img')
new_K[1, 2] = cv2.getTrackbarPos('cy', 'img')

roll = rollDeg * np.pi / 180
pitch = -pitchDeg * np.pi / 180

# ******** 我对以下几行代码存疑 **********  
# Rotation matrices for roll (rotX) and pitch (rotY)
rotX = np.array([[1, 0, 0],
                 [0, np.cos(roll), -np.sin(roll)],
                 [0, np.sin(roll), np.cos(roll)]])

rotY = np.array([[np.cos(pitch), 0, np.sin(pitch)],
                 [0, 1, 0],
                 [-np.sin(pitch), 0, np.cos(pitch)]])

# Combined rotation matrix
rotation = rotX @ rotY

# Calculate rotated intrinsic matrix
K_rotated = rotation @ K

# Generate undistortion map
mapx, mapy = cv2.fisheye.initUndistortRectifyMap(K_rotated, D, rotation, new_K, (1280, 720), cv2.CV_16SC2)
# ********         **********  

# Remap input image using undistortion map
undistorted_image = cv2.remap(input_image, mapx, mapy, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)

# Display undistorted image
cv2.imshow('img', undistorted_image)
#cv2.imwrite('undistortedImg.jpg', undistorted_image)

# Load input image
input_image = cv2.imread("distortedImg.png")

# Create named window and trackbars
cv2.namedWindow('img', cv2.WINDOW_NORMAL)
cv2.createTrackbar('roll', 'img', int(rollDeg), 89, lambda x: update_image())
cv2.createTrackbar('pitch', 'img', int(pitchDeg), 89, lambda x: update_image())
cv2.createTrackbar('fx', 'img', fx_init, 500, lambda x: update_image())
cv2.createTrackbar('fy', 'img', fy_init, 500, lambda x: update_image())
cv2.createTrackbar('cx', 'img', cx_init, 1920, lambda x: update_image())
cv2.createTrackbar('cy', 'img', cy_init, 1080, lambda x: update_image())

# Initialize undistorted image
update_image()

# Wait for key press to exit
cv2.waitKey(0)
cv2.destroyAllWindows()

你存疑的部分有3个核心错误：

1. 旋转矩阵方向与坐标系不匹配：OpenCV相机坐标系定义为X轴向右、Y轴向下、Z轴向前，你当前的rotX/rotY旋转方向和姿态需求完全颠倒，且组合顺序错误。
2. 错误旋转内参矩阵：K_rotated = rotation @ K完全不合理，内参是相机固有属性，不会随姿态旋转改变，旋转应该作用于外参的旋转部分，而非内参。
3. initUndistortRectifyMap参数逻辑错误：第三个参数R是原始相机到目标相机的旋转矩阵，你传入的是相机自身的旋转，逻辑完全颠倒。

import cv2
import numpy as np

# 原始相机姿态：左视图下向前轴下倾40°，俯视图下向前轴左倾30°
# 目标是转到俯视，需要抵消原姿态并调整角度
target_pitch_deg = 40  # 抵消原下倾的角度，可增大获得更陡俯视
target_yaw_deg = 30    # 抵消原左倾的角度

# Camera calibration parameters
K = np.array([[413.4459003089536, 0.0, 625.4657090067509],
              [0.0, 414.5602416017697, 393.99252903112915],
              [0.0, 0.0, 1.0]])

D = np.array([[0.2422953376828772],
              [0.5059306403608751],
              [-0.48731143899233426],
              [0.17754105229960523]])

# 初始化优化后的新内参矩阵
new_K = cv2.fisheye.estimateNewCameraMatrixForUndistortRectify(K, D, (1280, 720), np.eye(3), balance=1, new_size=(1280, 720), fov_scale=1)

# 滑块初始值用优化后的内参，而非自定义值
fx_init = int(new_K[0,0])
fy_init = int(new_K[1,1])
cx_init = int(new_K[0,2])
cy_init = int(new_K[1,2])

# 提前加载图像
input_image = cv2.imread("distortedImg.png")

def update_image():
    # 获取滑块参数
    pitch_deg = cv2.getTrackbarPos('pitch', 'img')
    yaw_deg = cv2.getTrackbarPos('yaw', 'img')
    new_K[0, 0] = cv2.getTrackbarPos('fx', 'img')
    new_K[1, 1] = cv2.getTrackbarPos('fy', 'img')
    new_K[0, 2] = cv2.getTrackbarPos('cx', 'img')
    new_K[1, 2] = cv2.getTrackbarPos('cy', 'img')

    # 转弧度
    pitch = np.deg2rad(pitch_deg)
    yaw = np.deg2rad(yaw_deg)

    # OpenCV坐标系下的旋转矩阵：先矫正左倾（绕Y轴），再矫正下倾（绕X轴）
    # 绕Y轴旋转：抵消原左倾，正方向为向右旋转
    rotY = np.array([
        [np.cos(yaw), 0, np.sin(yaw)],
        [0, 1, 0],
        [-np.sin(yaw), 0, np.cos(yaw)]
    ])
    # 绕X轴旋转：抵消原下倾，正方向为向上旋转（获得俯视）
    rotX = np.array([
        [1, 0, 0],
        [0, np.cos(pitch), -np.sin(pitch)],
        [0, np.sin(pitch), np.cos(pitch)]
    ])

    # 组合旋转矩阵：先绕Y再绕X
    R = rotX @ rotY

    # 生成矫正映射：使用原始内参K，R是原始到目标的旋转矩阵
    mapx, mapy = cv2.fisheye.initUndistortRectifyMap(K, D, R, new_K, (1280, 720), cv2.CV_16SC2)

    # 重映射并显示
    undistorted_image = cv2.remap(input_image, mapx, mapy, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)
    cv2.imshow('img', undistorted_image)

# 创建窗口和滑块
cv2.namedWindow('img', cv2.WINDOW_NORMAL)
cv2.createTrackbar('pitch', 'img', target_pitch_deg, 90, lambda x: update_image())
cv2.createTrackbar('yaw', 'img', target_yaw_deg, 60, lambda x: update_image())
cv2.createTrackbar('fx', 'img', fx_init, 800, lambda x: update_image())
cv2.createTrackbar('fy', 'img', fy_init, 800, lambda x: update_image())
cv2.createTrackbar('cx', 'img', cx_init, 1920, lambda x: update_image())
cv2.createTrackbar('cy', 'img', cy_init, 1080, lambda x: update_image())

# 初始化显示
update_image()

cv2.waitKey(0)
cv2.destroyAllWindows()

1. 旋转逻辑对齐坐标系：
   • 原相机左倾是绕Y轴负方向旋转，矫正时用正Y轴旋转抵消；原相机下倾是绕X轴负方向旋转，用正X轴旋转获得俯视。
2. 移除内参旋转操作：内参矩阵K是相机固有属性，直接传入initUndistortRectifyMap即可。
3. 修正参数命名：把原roll改为yaw，左倾属于偏航（绕Y轴）而非滚转（绕Z轴），避免概念混淆。
4. 滑块初始值优化：用estimateNewCameraMatrixForUndistortRectify生成的优化内参作为初始值，而非自定义的230，提升初始矫正效果。

内容的提问来源于stack exchange，提问作者Jai