基于OpenCV C++：从配准图像与单应/基础矩阵生成深度图的方法求助

First, let's clarify a critical point: homography alone can't generate a depth map unless your scene is entirely planar (no depth variation) or the camera only rotated without any translation. For true depth estimation from stereo images, you'll need to use the fundamental matrix alongside camera intrinsic parameters to recover camera pose, then compute dense disparity, and finally convert disparity to depth.

Here's the step-by-step breakdown to get from your current state to a depth map:

1. Recover Camera Pose from Fundamental Matrix #

To compute depth, you need to know the relative rotation (R) and translation (t) between your two cameras. This requires:

• Camera Intrinsic Matrices (K1, K2): If you haven't calibrated your camera, use OpenCV's calibrateCamera function with a chessboard pattern first. Intrinsics include focal length, principal point, and distortion coefficients (you’ve already handled radial distortion, so you can skip that part here).
• Essential Matrix Calculation: Derive the essential matrix E from the fundamental matrix F and intrinsics:

  Mat E = K2.t() * F * K1;
  

• Recover Rotation and Translation: Use OpenCV's recoverPose to extract valid R and t from E and your matched points:

  vector<Point2f> points1, points2;
  // Populate points1 and points2 with your good match coordinates
  Mat R, t;
  int inliers = recoverPose(E, points1, points2, K1, noArray(), R, t);
  

2. Stereo Rectification (Critical for Dense Matching) #

Before computing dense disparity, rectify both images so epipolar lines are horizontal—this simplifies pixel matching drastically:

• Compute rectification transforms and projection matrices with stereoRectify:

  Mat R1, R2, P1, P2, Q;
  stereoRectify(K1, noArray(), K2, noArray(), imageSize, R, t, R1, R2, P1, P2, Q, CALIB_ZERO_DISPARITY);
  

• Apply rectification to both images using initUndistortRectifyMap and remap:

  Mat map1x, map1y, map2x, map2y;
  initUndistortRectifyMap(K1, noArray(), R1, P1, imageSize, CV_32FC1, map1x, map1y);
  initUndistortRectifyMap(K2, noArray(), R2, P2, imageSize, CV_32FC1, map2x, map2y);
  
  Mat rectified1, rectified2;
  remap(image1, rectified1, map1x, map1y, INTER_LINEAR);
  remap(image2, rectified2, map2x, map2y, INTER_LINEAR);
  

3. Compute Dense Disparity Map #

Use OpenCV's StereoSGBM (more accurate than StereoBM) to find pixel-wise correspondences between rectified images:

Ptr<StereoSGBM> sgbm = StereoSGBM::create(
    0,                  // Minimum disparity
    16,                 // Number of disparities (must be multiple of 16)
    3,                  // Block size
    8*3*3,              // P1 parameter (smoothness penalty for adjacent pixels)
    32*3*3,             // P2 parameter (stricter penalty for larger disparity differences)
    1,                  // Disp12MaxDiff
    63,                 // PreFilterCap
    10,                 // UniquenessRatio
    100,                // SpeckleWindowSize
    32,                 // SpeckleRange
    StereoSGBM::MODE_SGBM_3WAY
);

Mat disparity;
sgbm->compute(rectified1, rectified2, disparity);

// Convert disparity to 32-bit float for depth calculation
Mat disp_float;
disparity.convertTo(disp_float, CV_32F, 1.0/16.0); // Divide by 16 to get actual disparity values

4. Convert Disparity to Depth #

Use either the reprojection matrix Q or manual formula to convert disparity to depth:

• Using Q (handles all calibration parameters automatically):

  Mat depth_map;
  reprojectImageTo3D(disp_float, depth_map, Q, true);
  // Extract the depth channel (third channel in the 3D map)
  Mat depth;
  extractChannel(depth_map, depth, 2);
  

• Manual calculation (if you have focal length f and baseline B in real units):

  float f = K1.at<double>(0,0); // Focal length in pixels
  float B = norm(t); // Baseline (ensure t is in real-world units; adjust if using normalized coordinates)
  Mat depth = (f * B) / disp_float;
  

Key Notes #

• Homography Limitation: If you relied on homography, depth will be uniform (only works for planar scenes or pure camera rotation). For scenes with depth variation, stick to the fundamental matrix workflow.
• Disparity Tuning: Adjust StereoSGBM parameters (e.g., number of disparities, block size) based on your images to reduce noise and improve matching accuracy.
• Calibration Importance: Without accurate intrinsics, your depth map will only show relative depth (not absolute real-world values).

内容的提问来源于stack exchange，提问作者Gandalf