AR场景下相机内外参转OpenGL模型视图与投影矩阵的技术咨询

Hey there! Let's walk through how to convert your camera's intrinsic/extrinsic matrices to OpenGL's model-view and projection matrices, plus solidify how to validate your extrinsic matrix correctly.

First, Understand Coordinate System Differences #

This is the most common pitfall—camera and OpenGL use different coordinate conventions:

• Camera Coordinate System: Z-axis points forward (toward the scene), Y-axis points downward, origin at the camera lens.
• OpenGL View Coordinate System: Z-axis points backward (away from the scene, toward the observer), Y-axis points upward, origin at the camera lens.
  You'll need to account for these flips in your matrix conversions.

Step 1: Convert Extrinsic Matrix to OpenGL Model-View Matrix #

Your extrinsic matrix E (usually a 4x4 homogeneous matrix [R | t]) defines the transformation from world coordinates to camera coordinates:
X_cam = E * X_world (where X is a 4x1 homogeneous point)

To get OpenGL's model-view matrix, we need to map world coordinates to OpenGL's view space. Here's how:

1. Build a coordinate flip matrix T to align camera space with OpenGL view space:

   T = [
       1  0  0  0,
       0 -1  0  0,  // Flip Y-axis
       0  0 -1  0,  // Flip Z-axis
       0  0  0  1
   ]
   

2. Compute the view matrix:
   The view matrix transforms world points to OpenGL view space. Since E maps world to camera space, we combine it with the flip matrix:
   M_view = T * E
   Wait—if your extrinsic matrix is defined as camera-to-world (instead of world-to-camera), you'll need to invert E first: M_view = T * inverse(E). Double-check your extrinsic's definition here!

   Alternatively, use OpenGL's glm::lookAt function (if you're using GLM) to construct the view matrix directly:

   • Camera position in world space: cam_pos = -transpose(R) * t (since t is the world origin in camera space)
   • Camera forward direction (world space): forward = vec3(R[0][2], R[1][2], R[2][2]) (Z-axis of camera)
   • Camera up direction (world space): up = vec3(-R[0][1], -R[1][1], -R[2][1]) (inverted Y-axis of camera)
     Then: M_view = glm::lookAt(cam_pos, cam_pos + forward, up)

Step 2: Convert Intrinsic Matrix to OpenGL Projection Matrix #

Your intrinsic matrix K (3x3) looks like this:

K = [
    fx  0  cx,
    0  fy  cy,
    0   0   1
]

Where fx/fy are focal lengths (in pixels), cx/cy are principal point coordinates (image center in pixels).

To convert this to OpenGL's 4x4 projection matrix:

1. Define your image dimensions (W, H) and near/far clip planes (near_plane, far_plane—these are distances from the camera in world units).
2. Construct the projection matrix using this formula:

   P_gl = [
       2*fx/W,    0,          (2*cx/W) - 1,   0,
       0,         -2*fy/H,    1 - (2*cy/H),   0,
       0,          0,         -(far+near)/(far-near),  -2*far*near/(far-near),
       0,          0,         -1,              0
   ]
   

   • The negative sign on 2*fy/H accounts for the Y-axis flip between camera and OpenGL.
   • The Z-axis transformation maps camera space's positive Z (forward) to OpenGL's negative Z (view space).

Step 3: Validate Your Extrinsic Matrix (Your Current Approach) #

You're already using P = K * E to project CAD points onto the image—great! Here's how to make this validation robust:

• Use homogeneous coordinates correctly: Extend your 3D world points to 4x1 (X_world = [X, Y, Z, 1]), multiply by E (4x4) to get camera space points, then multiply by K (3x3) to get [u, v, w]. Normalize to get pixel coordinates: (u/w, v/w).
• Check feature point alignment: Pick distinct, easy-to-identify vertices from your CAD model (e.g., corners of a cube). Project them using P, then compare their pixel positions to the corresponding points in your image. A small error (1-3 pixels) means your extrinsic is correct.
• Debug common issues:
  • If projections are mirrored or flipped: Double-check the coordinate flip matrix T or the sign of your rotation/translation in E.
  • If scale is off: Ensure your extrinsic matrix uses the same unit system as your CAD model (e.g., meters vs. millimeters—though projection is scale-invariant, misalignment here can cause issues with depth).
  • If rotation is wrong: Verify that your world-camera correspondences are correctly paired (e.g., you didn't mix up X/Y axes when collecting points).

Quick Tips for Smooth Integration #

• Always verify that your rotation matrix R is orthogonal (transpose equals inverse, determinant = 1)—a non-orthogonal R will break projections.
• If your camera has distortion parameters, apply distortion to the projected (u/w, v/w) coordinates before overlaying (most camera calibration tools give you distortion coefficients for this).
• Test with a simple CAD model first (like a cube) before moving to complex models—this makes it easier to spot errors.

内容的提问来源于stack exchange，提问作者Wooni