MongoDB新手求助：如何将现有数据封装为嵌入式文档？

Hey there! No worries at all—this is a totally common scenario when restructuring MongoDB data, especially as you're getting started. Let's walk through your options, starting with the in-place update (since you prefer that) and then covering the export/import approach.

In-place updates are great because you don't have to deal with large export files or switching collections. The core idea is to wrap all existing fields into an embedded document (let's call it data for this example) and then remove the original top-level fields (except _id, since you can't delete that).

Basic Update Command #

First, here's the core update logic using updateMany. It uses aggregation pipeline stages to dynamically capture all fields, so you don't have to manually list 409 fields:

db.yourCollection.updateMany(
  {}, // Match all documents
  [
    // Step 1: Wrap the entire document into the `data` embedded field
    { $set: { data: "$$ROOT" } },
    // Step 2: Remove all top-level fields except `_id`
    {
      $unset: {
        $expr: {
          $objectToArray: {
            $filter: {
              input: { $objectToArray: "$$ROOT" },
              cond: { $ne: ["$$this.k", "_id"] }
            }
          }
        }
      }
    }
  ]
)

Batch Processing for Large Datasets #

With 10 million documents, running the above command all at once might cause timeouts or strain your database. Instead, process documents in batches to keep things manageable:

const batchSize = 1000;
let processed = 0;
const totalDocs = db.yourCollection.countDocuments({});

print(`Starting batch processing of ${totalDocs} documents...`);

while (processed < totalDocs) {
  const updateResult = db.yourCollection.updateMany(
    { data: { $exists: false } }, // Only update unprocessed docs
    [
      { $set: { data: "$$ROOT" } },
      {
        $unset: {
          $expr: {
            $objectToArray: {
              $filter: {
                input: { $objectToArray: "$$ROOT" },
                cond: { $ne: ["$$this.k", "_id"] }
              }
            }
          }
        }
      }
    ],
    { limit: batchSize }
  );

  processed += updateResult.modifiedCount;
  print(`Processed ${processed}/${totalDocs} documents`);
  
  // Optional: Add a short delay to avoid overwhelming the DB
  sleep(100);
}

print("Batch processing complete!");

Key Notes #

• Backup First: Always back up your collection before running bulk updates—once you unset those top-level fields, recovering them is hard without a backup.
• Version Check: This uses $expr in $unset, which requires MongoDB 4.2 or later. If you're on an older version, you'll need to manually list the fields to unset (not ideal for 409 fields, so upgrading might be worth it).
• Index Considerations: If you have indexes on the top-level fields, you'll want to drop them after the update (since those fields no longer exist) and add indexes to the embedded data fields if needed.

If in-place updates aren't feasible (e.g., you're worried about impacting live traffic, or your MongoDB version doesn't support the pipeline features), this approach works well.

Step 1: Export Original Data #

Use mongoexport to dump your collection to a JSON file:

mongoexport --db yourDatabaseName --collection yourCollectionName --out original_data.json

Step 2: Transform the Data #

You can use a tool like jq (a lightweight JSON processor) to quickly wrap all fields into an embedded document. This is way more efficient than writing a custom script for large datasets:

# This creates a new file where each document has `_id` at the top level, and all other fields inside `data`
cat original_data.json | jq '{_id: ._id, data: del(._id)}' > transformed_data.json

If you don't have jq installed, you could use a Node.js/Python script, but jq is the fastest option here.

Step 3: Import the Transformed Data #

You can either overwrite the original collection (make sure you have a backup!) or import into a new collection:

# Option 1: Overwrite the original collection (backup first!)
mongo --eval "db.yourCollectionName.deleteMany({})" yourDatabaseName
mongoimport --db yourDatabaseName --collection yourCollectionName --file transformed_data.json

# Option 2: Import into a new collection (safer for live environments)
mongoimport --db yourDatabaseName --collection newCollectionName --file transformed_data.json

Key Notes #

• Data Types: mongoexport converts MongoDB-specific types (like ObjectId, ISODate) to JSON strings, but mongoimport will automatically convert them back to the correct types—no extra work needed.
• Disk Space: 10 million documents with 410 fields will create a very large JSON file—make sure you have enough disk space before exporting.
• Downtime: If you're overwriting the original collection, you'll need to plan for some downtime (or use a new collection and switch over once the import is done).

内容的提问来源于stack exchange，提问作者chanman803