3D点投影至图像平面失败求助：世界-相机-像素坐标转换问题

Hey, I’ve been through similar headaches with 3D projection before—let’s break down the likely issues causing your misalignment and negative coordinates:

1. Coordinate System Mismatch (Critical!) #

First, your defined coordinate system x=depth, z=up, y=horizontal directly conflicts with OpenCV’s default camera coordinate system, where z=depth (camera optical axis). This is almost certainly the biggest culprit behind your wrong projections.

When you do things like:

xt = mpoint3.at<double>(0) / mpoint3.at<double>(2);

You’re dividing by the z-axis (your up axis) instead of the x-axis (your depth axis). For your custom coordinate system, the correct normalized image coordinates should be:

// Normalize using depth axis (x) as the denominator
double xt = mpoint3.at<double>(1) / mpoint3.at<double>(0);  // y (horizontal) / depth x
double yt = mpoint3.at<double>(2) / mpoint3.at<double>(0);  // z (up) / depth x
double u = xt * fx + cx1;
double v = yt * fy + cy1;

If you use OpenCV’s built-in functions like fisheye::projectPoints, you’ll need to either:

• Convert your 3D points and camera pose to match OpenCV’s z-depth system (swap x and z axes for points and adjust rotation matrices accordingly)
• Rewrite your rotation matrices to align your x-depth axis with OpenCV’s expected z-depth axis.

2. Incorrect World-to-Camera Transform Order #

Your current code applies rotation before translation, which is backwards. The correct formula for world-to-camera coordinates is:
$$P_{cam} = R \times (P_{world} - T)$$
Where $T$ is the camera’s position in world space.

Your code does:

mpoint3 = mpoint3 * rotationm; // Rotate first
mpoint3 = mpoint3 - position;  // Translate second

Which is mathematically wrong. Fix it by reversing the order:

// First, compute point relative to camera position
cv::Mat mpoint3_rel = mpoint3 - position;
// Then apply rotation (keep row-vector multiplication since you defined mpoint3 as 1x3)
mpoint3 = mpoint3_rel * rotationm;

Or, for more standard column-vector notation (preferred in computer vision):

cv::Mat mpoint3_col(3, 1, CV_64F, {-455, -150, 0});
cv::Mat position_col(3, 1, CV_64F, {-50, 0, 100});
cv::Mat mpoint3_rel = mpoint3_col - position_col;
cv::Mat mpoint3_cam = rotationm * mpoint3_rel;

3. Rotation Matrix Multiplication Order #

You’re combining rotations as rotz * roty * rotx, which applies rotx first, then roty, then rotz. But camera pose rotations are usually specified in the order yaw → pitch → roll (not roll → pitch → yaw). If your yp (roll), thet (pitch), k (yaw) are given in that order, your rotation matrix should be:

cv::Mat rotationm = rotx * roty * rotz;

Always double-check the rotation order against how your camera’s pose parameters are defined—mixing this up will cause major orientation shifts.

4. Wrong Parameters for fisheye::projectPoints #

When using this function, note that:

• The tvec parameter is not the camera’s world position—it’s the negative translation vector from the world origin to the camera, i.e., tvec = -position_col.
• inputpoints should be a vector of 3D world points (or a Nx1x3 Mat), not camera-space points.
• Ensure your rotation vector recv2 is derived from the correct world-to-camera rotation matrix (if your matrix is camera-to-world, you need to transpose it first).

Here’s a corrected call example:

std::vector<cv::Point3d> inputpoints = {{-455, -150, 0}};
std::vector<cv::Point2d> outputpoints;
cv::Mat rvec, tvec;
cv::Rodrigues(rotationm, rvec);
tvec = -position_col; // Critical: negative camera position
cv::fisheye::projectPoints(inputpoints, outputpoints, rvec, tvec, mycameraMatrix, mydiscoff);

Quick Validation Steps #

1. Test with a trivial point: Project the camera’s own position ((-50,0,100))—it should map to the image center (cx, cy) if everything is set up right.
2. Verify your rotation matrices by applying them to axis-aligned points (e.g., (1,0,0)) and checking if the result matches your expected orientation.
3. Print intermediate values (relative point, rotated point, normalized coordinates) to spot where the numbers go wrong.

内容的提问来源于stack exchange，提问作者YAMI