无靶标非重叠立体相机标定方案咨询及相关技术疑问

Great set of questions—non-overlapping targetless stereo calibration is a tricky problem because you can't rely on shared visual features between the two cameras directly. Let's walk through each of your queries with practical, actionable answers:

1. How to perform calibration for non-overlapping targetless stereo camera systems? #

The core challenge here is establishing a common reference between the two cameras, since they don't see the same parts of the scene at the same time. Here are the most reliable approaches:

• Structure-from-Motion (SfM) with shared motion: If you can move either a common object (any distinct object works) through both cameras' fields of view (sequentially, not simultaneously), run SfM on each camera's footage to build separate 3D maps. Then, use the shared 3D points from the moving object to align the two maps—this alignment gives you the relative extrinsic parameters between the cameras.
• Rigid rig motion in a static scene: If the two cameras are fixed together as a rigid rig, move the entire rig around a static scene with plenty of visual features. Each camera will generate its own SfM trajectory. Since the rig is rigid, the relative pose between the cameras stays constant; you can compute this fixed transform by comparing corresponding pose pairs from each camera's trajectory.
• Sensor fusion with auxiliary hardware: If you have IMUs, LiDAR, or GPS attached to the rig, use these sensors to get absolute or relative pose estimates for each camera. The difference between these poses gives you the stereo extrinsic parameters.

2. Can we use visual odometry (like ORB-SLAM) to compute trajectories of two rigidly fixed cameras, then use hand-eye calibration to get extrinsic parameters? #

Absolutely—this is one of the most practical and robust methods for this scenario. Here's why it works:

• ORB-SLAM (or any robust VO/SLAM system) outputs the pose of each camera relative to its starting frame for every frame. Since the cameras are rigidly attached, the transformation between Camera A and Camera B (T_AB) is constant across all motion.
• Hand-eye calibration (the classic AX=XB problem) is designed exactly for this:
  • A = incremental pose of Camera A between two consecutive frames
  • B = incremental pose of Camera B between the same two frames
  • X = fixed transform from Camera A to Camera B
    Collect a dozen or more A/B pairs from the VO trajectories, then solve the AX=XB system to get X—your stereo extrinsic parameters.

3. If the above approach is feasible, how to map the trajectory transformation relationships to gripper->base and target->camera transformations? #

Let's translate the stereo rig scenario to the hand-eye calibration framework clearly:

• In standard hand-eye calibration, we solve AX=XB where A is the base-to-base motion (robot arm moving), B is the camera-to-camera motion, and X is the fixed gripper-to-camera transform.
• For your stereo rig:
  • Let T_A_world = pose of Camera A in the world frame (from ORB-SLAM) at time t
  • Let T_B_world = pose of Camera B in the world frame at time t
  • The stereo transform you want is T_B_A (Camera B relative to Camera A), which equals T_B_world * T_A_world.inverse()
  • To fit to AX=XB:
    • A = T_A_world_t1 * T_A_world_t0.inverse() (Camera A's pose change between frames t0 and t1)
    • B = T_B_world_t1 * T_B_world_t0.inverse() (Camera B's pose change between the same frames)
    • Solve A*X = X*B for X = T_B_A
• If you're working with a robot arm setup:
  • T_gripper_base would be the arm's end effector pose relative to the base (from robot joint data)
  • T_target_camera can be computed once you have T_B_A: reconstruct the target in one camera's frame, then transform it to the other camera's frame using the calibrated stereo extrinsic.

4. Are there other applications of this algorithm? #

This combination of VO/SLAM + hand-eye calibration is surprisingly versatile beyond stereo rigs:

• Multi-camera surround calibration: Calibrate 3+ non-overlapping cameras (like in car surround-view systems) by treating each pair as a stereo rig and extending the same method.
• Robot arm-camera calibration: If a camera is mounted on a robot arm but doesn't overlap with the end effector, use VO to track the camera's motion as you move the arm, then solve hand-eye to get the camera-to-gripper transform.
• Sensor cross-calibration: Calibrate the transform between a camera and an IMU/LiDAR (no overlapping target needed) by using VO for camera poses and sensor data for motion estimates, then applying a hand-eye-style solve.
• Large-scale scene reconstruction: Merge 3D maps from non-overlapping cameras (e.g., for indoor mapping) using the calibrated extrinsic parameters to create a unified, full-scene model.

5. If hand-eye calibration can't be used, what are feasible suggestions for targetless non-overlapping stereo camera calibration? #

If hand-eye isn't an option (e.g., you can't move the rig, or VO fails in the scene), try these alternatives:

• Dynamic object tracking: Track a distinct moving object (person, car, even a handheld marker-free object) through both cameras' fields of view. Use robust trackers (like KLT or DeepSORT) to follow the object's 2D features across frames, then triangulate these features over time to estimate the relative camera pose.
• Scene structure alignment: Run SfM on each camera's footage to build separate 3D point clouds of the scene. Use point cloud registration techniques (ICP or feature-based matching) to align the two clouds—this alignment transform is your stereo extrinsic parameters. Works best if the scene has unique structural features (e.g., architectural elements).
• Known scene constraints: If you have prior knowledge about the scene (e.g., certain objects are on a flat plane, or have known dimensions), use these constraints to compute each camera's pose relative to the constraint, then derive the relative camera-to-camera transform. For example, align two camera poses relative to a shared wall plane.
• Temporary portable targets: Even if you can't use a fixed target, use a portable calibration target (like a chessboard) that you move to be visible by each camera separately. Capture images of the target with Camera A, then move it to Camera B's field of view. Compute each camera's pose relative to the target, then calculate the stereo transform from those two poses.

内容的提问来源于stack exchange，提问作者Sandeep Menon