Mongoose批量Upsert及条件更新实现问题（含代码报错排查）

Hey there! Let's tackle your two main issues: the frustrating _id field change error in your bulk upsert, and building an efficient sync flow that handles both upserts and cleaning up old inventory records.

What's Causing the _id Error? #

The root of the problem is how you're preparing the update document:

• When you run new Item(data), Mongoose automatically generates a brand new _id for that instance—even if your original data doesn't include one.
• When you pass this full instance to updateOne(newItem), MongoDB tries to overwrite the existing document's _id with this new value. Since MongoDB doesn't allow modifying the _id field once a document is created, this throws an error.

You don't need to create a Mongoose model instance for bulk operations here; we can work directly with the raw data to avoid this issue entirely.

Efficient Bulk Upsert + Old Record Cleanup Solution #

Here's a revised implementation that fixes the _id error, handles proper upserts (updating attributes like name when they change), and cleans up old inventory items no longer present in the latest API data:

import Item from "../models/item.model";
import _ from 'lodash';

async function storeInventory(items) {
  // Step 1: Validate incoming items and collect unique identifiers
  const validItems = _.filter(items, (data) => data !== null);
  const currentItemIdentifiers = validItems.map(item => ({
    classId: item.classId,
    instanceId: item.instanceId
  }));

  if (validItems.length === 0) {
    // Optional: If no items are present, you could delete all inventory here
    // await Item.deleteMany({});
    return;
  }

  // Use a transaction to ensure atomicity (either all operations succeed or none)
  const session = await Item.startSession();
  session.startTransaction();

  try {
    // Step 2: Perform bulk upsert operation
    const bulkUpdate = Item.collection.initializeUnorderedBulkOp({ session });
    
    _.forEach(validItems, (data) => {
      // Explicitly exclude _id from the update to avoid conflicts
      const updateDoc = { $set: _.omit(data, '_id') };
      
      bulkUpdate.find({ 
        classId: data.classId, 
        instanceId: data.instanceId 
      })
      .upsert()
      .updateOne(updateDoc);
    });

    await bulkUpdate.execute();

    // Step 3: Delete old inventory items not in the latest API data
    await Item.deleteMany({
      $nor: currentItemIdentifiers
    }, { session });

    await session.commitTransaction();
    console.log('Inventory sync completed successfully');
  } catch (error) {
    await session.abortTransaction();
    console.error('Inventory sync failed:', error);
    throw error;
  } finally {
    session.endSession();
  }
}

Key Improvements Explained: #

• Fixes _id Errors: We use _.omit(data, '_id') to ensure we never pass a new _id to the update operation. The $set operator only updates the relevant fields from your raw API data, leaving the existing _id untouched.
• Atomic Sync: Wrapping upserts and deletes in a transaction ensures data consistency—you won't end up with half-updated inventory if something goes wrong mid-sync.
• Efficient Cleanup: The $nor operator matches all documents that don't belong to the current inventory list, deleting old records in a single query instead of multiple operations.
• Performance: Unordered bulk operations are ideal for your 600-2500 item range, as they process operations in parallel without stopping on individual failures.

Additional Tips: #

• Indexing: Add a compound unique index on classId and instanceId to speed up find operations in bulk upserts and the delete query. Add this to your item.model.js schema:

  ItemSchema.index({ classId: 1, instanceId: 1 }, { unique: true });
  

• Null Handling: We filter out null items upfront to skip unnecessary operations.
• Error Resilience: The transaction rollback ensures you don't leave your database in an inconsistent state if an error occurs.

内容的提问来源于stack exchange，提问作者Crazy Redd