OpenCV_contrib中SfM模块reconstruct()函数InputArrayOfArrays参数疑问

Hey there! Let me break down your questions about the points2d parameter and feature matching for the SfM reconstruct() function, based on my hands-on experience with the module.

First: Do you need to convert Mat data to a vector of vectors? #

Nope, you don’t have to jump through hoops converting formats—OpenCV’s InputArrayOfArrays is designed to be flexible. It accepts multiple types of input as long as the structure matches the requirement:

• A vector<vector<Point2f>>: Where the outer vector represents each image, and the inner vector holds the 2D points detected/matched in that image.
• A vector<Mat>: Each Mat should be an N×2 matrix of type CV_32F, where each row is a (x, y) coordinate of a 2D point for that image.
• Even a raw array of Mat objects works, though vectors are more convenient in practice.

The key is consistency: each entry in the outer container (whether vector or array) corresponds to one image, and the points inside each entry map to the same 3D point across all images. OpenCV handles the conversion between these formats internally, so pick the one that fits your workflow best.

Second: Do you need a feature matcher? #

Yes, absolutely—this is non-negotiable. The reconstruct() function relies on having corresponding 2D points across multiple images (i.e., points that are projections of the same 3D point in the real world). Here’s the typical workflow to get valid points2d data:

1. Detect features in every image (using SIFT, ORB, SURF, or another feature detector).
2. Match features across image pairs (using BFMatcher, FlannBasedMatcher, etc.).
3. Filter matches to remove outliers (using techniques like RANSAC with findFundamentalMat or findHomography).
4. Consolidate corresponding points: For all images, collect the 2D points such that the k-th point in every image’s list is the projection of the same 3D point. This is how points2d needs to be structured.

Quick code snippet to illustrate #

Here’s a simplified example of how you might prepare points2d:

// Assume we have already detected features and filtered matches across all images
vector<vector<Point2f>> points2d;
int num_images = ...; // Number of images in your sequence

// For each image, collect its corresponding 2D points
for (int img_idx = 0; img_idx < num_images; ++img_idx) {
    vector<Point2f> current_img_points;
    // Populate current_img_points with the matched points for this image
    // (Make sure the order matches across all images!)
    points2d.push_back(current_img_points);
}

// Alternatively, using vector<Mat>
vector<Mat> points2d_mats;
for (auto& img_points : points2d) {
    Mat mat(img_points.size(), 2, CV_32F);
    for (int i = 0; i < img_points.size(); ++i) {
        mat.at<float>(i, 0) = img_points[i].x;
        mat.at<float>(i, 1) = img_points[i].y;
    }
    points2d_mats.push_back(mat);
}

// Now call reconstruct with either points2d or points2d_mats
Mat camera_matrix = ...; // Your calibrated camera intrinsic matrix
vector<Mat> rotation_mats, translation_vecs;
vector<Point3f> reconstructed_3d_points;

sfm::reconstruct(points2d, rotation_mats, translation_vecs, camera_matrix, reconstructed_3d_points);

A quick note on edge cases #

If some images don’t have a projection of a particular 3D point, you can use the optional mask parameter (a vector<vector<bool>>) to mark which points are valid for each image. But in most cases, it’s easier to pre-process your matches to only include points that are visible in all (or most) images first.

内容的提问来源于stack exchange，提问作者AndrewD