Node.js从Minio获取JSON文件存入MongoDB(Mongoose)遇Buffer问题求助

Hey there! Let's break down what's going wrong with your code and fix it so you can properly fetch JSON from Minio and save it to MongoDB with Mongoose.

What's Wrong with Your Current Code #

Your code has three key issues:

1. Partial data handling: The data event fires in chunks, and you're overwriting miniData with each chunk instead of collecting all parts. This means you'll only get the last piece of your file, not the full content.
2. Incorrect Buffer conversion: Calling JSON.stringify(chunk) on a Buffer doesn't parse the actual JSON content—it just converts the Buffer object itself into a string (resulting in that {"type":"Buffer","data":[...]} output you're seeing).
3. Async timing mismatch: You're returning miniData immediately after setting up event listeners, but the data hasn't been fetched yet. The callback-based approach means your return happens before the data events even fire.

The Solution: Collect All Chunks & Handle Async Properly #

We'll wrap the Minio stream in a Promise to handle async cleanly, collect all chunks, merge them into a complete Buffer, then parse it into valid JSON.

Step 1: Create a Reusable Function to Fetch & Parse JSON from Minio

function fetchMinioJson(bucketName, objectKey) {
  return new Promise((resolve, reject) => {
    const chunks = [];
    
    minioClient.getObject(bucketName, objectKey, (err, dataStream) => {
      if (err) {
        return reject(new Error(`Failed to get object: ${err.message}`));
      }

      // Collect each data chunk as it arrives
      dataStream.on('data', (chunk) => {
        chunks.push(chunk);
      });

      // Process full data once the stream ends
      dataStream.on('end', () => {
        try {
          // Merge all chunks into a single Buffer, convert to UTF-8 string
          const fullDataString = Buffer.concat(chunks).toString('utf8');
          // Parse the string into a valid JSON object
          const jsonData = JSON.parse(fullDataString);
          resolve(jsonData);
        } catch (parseError) {
          reject(new Error(`Failed to parse JSON: ${parseError.message}`));
        }
      });

      // Handle any stream errors
      dataStream.on('error', (streamError) => {
        reject(streamError);
      });
    });
  });
}

Step 2: Use the Function to Save Data to MongoDB

Now you can use async/await to fetch the JSON and save it with Mongoose seamlessly:

async function saveMinioDataToMongo(fileData) {
  try {
    const objectKey = `${fileData.userId}${fileData.fileName}`;
    // Fetch parsed JSON from Minio
    const jsonData = await fetchMinioJson(fileData.type, objectKey);
    
    // Save to MongoDB using your Mongoose model
    await YourMongooseModel.create(jsonData);
    
    console.log('Successfully saved data to MongoDB!');
  } catch (error) {
    console.error('Error during process:', error);
  }
}

// Execute the function with your fileData
saveMinioDataToMongo(fileData);

Why This Works #

• Promise wrapping: This lets us use async/await to ensure we have the full JSON data before trying to save it to MongoDB, avoiding race conditions.
• Chunk collection: By pushing every chunk to an array and merging them with Buffer.concat, we get the complete file content instead of partial data.
• Proper JSON parsing: Converting the merged Buffer to a UTF-8 string first, then parsing it with JSON.parse, gives us a valid JavaScript object that Mongoose can work with directly.

内容的提问来源于stack exchange，提问作者Caner Ünver