基于已知相机位姿与标定参数，从图像投影估计三维直线的方法咨询

Hey there! Let's tackle your 3D line estimation problem head-on. First off, the approach you're thinking of—using planes formed by each camera's center and the corresponding image line, then finding their intersection to get the 3D line—is totally feasible, and it's actually a standard method in multi-view geometry. Let's break down how it works and how to implement it.

Here's the key insight: Any 3D line, when projected onto a camera's image plane, forms an image line. Every point on the 3D line lies on a ray from the camera's optical center to its projection on the image line. All these rays lie within a single plane that contains both the camera's optical center and the entire 3D line. So if you compute this plane for every camera view, their mutual intersection will be exactly the 3D line you're trying to estimate—assuming the planes aren't parallel (which they won't be with diverse camera angles).

Let's walk through the concrete steps, assuming you have:

• Camera intrinsic matrix K (calibrated)
• Camera extrinsic parameters [R_i | t_i] for each view i (world-to-camera rotation and translation)
• Image line in each view, represented as a_i x + b_i y + c_i = 0 (pixel coordinates x,y)

1. Compute the 3D Plane for Each View #

First, convert each image line into a 3D plane (in world coordinates):

• Step 1a: Convert image line to camera-space rays
  Pick two distinct points on the image line, e.g., P1 = (u1, v1) and P2 = (u2, v2). Convert these pixel coordinates to normalized camera-space directions using the inverse intrinsic matrix:

  p1 = K⁻¹ * [u1, v1, 1]^T
  p2 = K⁻¹ * [u2, v2, 1]^T
  

  These p1 and p2 are unit (or scaled) vectors pointing from the camera's optical center (origin in camera space) toward the 3D line.

• Step 1b: Compute plane normal in camera space
  The plane containing the camera center and the two rays has a normal vector equal to the cross product of p1 and p2:

  n_c = p1 × p2
  

  The camera-space plane equation is n_c · (X_c, Y_c, Z_c) = 0 (since it passes through the camera center at (0,0,0)).

• Step 1c: Convert plane to world coordinates
  The camera's optical center in world space is C_w = -R_i^T * t_i. The plane normal in world space is n_w = R_i^T * n_c (rotating the camera-space normal back to world space). The world-space plane equation becomes:

  n_w · (X_w - C_w) = 0
  

  Expanding this gives the standard plane form A_i X + B_i Y + C_i Z + D_i = 0, where D_i = -n_w · C_w.

2. Estimate the 3D Line from Multiple Planes #

With N planes (N ≥ 2), you need to find their common intersection line. For real-world data (with noise and imperfect image lines), use a least-squares fit instead of just intersecting two planes (which would be noisy):

• Step 2a: Find the line's direction vector
  The direction vector L of the 3D line must be perpendicular to every plane's normal n_w_i (since the line lies in all planes). This gives a system of homogeneous equations:

  n_w_1 · L = 0
  n_w_2 · L = 0
  ...
  n_w_N · L = 0
  

  To solve this, stack all n_w_i into a matrix N_mat, then perform SVD decomposition on N_mat. The singular vector corresponding to the smallest singular value is your direction vector L.

• Step 2b: Find a point on the line
  We need a point X0 that lies on all planes (or as close as possible in least-squares terms). From the plane equation n_w_i · X0 = n_w_i · C_w_i, we get a linear system. Stack these into a matrix A (rows are n_w_i) and vector b (entries are n_w_i · C_w_i), then solve A X0 = b using least squares:

  X0 = (A^T A)⁻¹ A^T b
  

• Step 2c: Parametrize the final line
  The 3D line can now be written as:

  X(w) = X0 + w * L
  

  where w is any real scalar.

• Image line accuracy: Use subpixel-accurate line detection (e.g., refined Hough transform or edge-based line fitting) to minimize errors in the image line parameters.
• Camera pose quality: Inaccurate R_i and t_i will skew your plane calculations. If your poses are estimated (not ground truth), ensure they're refined with bundle adjustment first.
• View diversity: Avoid having all cameras lie in a single plane relative to the 3D line—this can make the plane system ill-conditioned. Use views with significant angular separation for better results.
• Outlier handling: If some views have incorrect image lines (e.g., partial occlusion leading to wrong line detection), use robust estimation techniques like RANSAC to discard outliers before fitting the line.

内容的提问来源于stack exchange，提问作者mojado