Node.js本地连接MongoDB遇socket关闭错误的排查与优化

Let’s break this down step by step, with a close look at your code since that’s where most of the scaling pain points are coming from.

1. Why does the "Error shutting down socket" happen? #

This error almost always traces back to connection exhaustion or resource limits when you hit that ~680 concurrent connection mark. Here’s the breakdown specific to your setup:

• Uncontrolled connection creation: Your File_Recipes function spins up a brand-new MongoDB client every single time it’s called (once per grocery store combo). Each client establishes its own set of connections to MongoDB, and when you run hundreds of these in parallel, you quickly hit either:
  • MongoDB’s default maximum connection limit (default is 1000, but 680 pushes close to that threshold—especially if other system processes are using connections too)
  • Operating system file handle limits (each MongoDB connection uses a file handle; most Linux systems default to 1024 handles per process, so 680 connections are straining this boundary)
• Unmanaged async operations: Your cursor.forEach calls async functions (adaptIngredientOne, adaptIngredientTwo) but doesn’t wait for them to complete. This means you’re spawning hundreds of concurrent database reads/writes on top of already excessive client connections, overwhelming both Node.js’s event loop and MongoDB’s ability to handle socket traffic.
• Connection leaks: You never properly close the MongoDB client after operations finish. Those idle connections hang around, eating up resources until the system has no choice but to shut sockets down abruptly.

2. How to configure Node.js/MongoClient/MongoDB to use your hardware effectively? #

Your Ryzen 7 1800 (8 cores/16 threads) and 32GB RAM have plenty of headroom—you just need to fix connection management and concurrency controls. Here’s what to do:

Step 1: Reuse a single MongoDB client (connection pool) #

Stop creating a new client for every grocery combo. Create one client at app startup and reuse it everywhere. This lets MongoDB’s built-in connection pool handle efficient connection reuse instead of spawning hundreds of redundant connections.

// Create a single client instance in a separate db.js file
const { MongoClient } = require('mongodb');
const url = 'your-mongodb-url';
const client = new MongoClient(url, {
  maxPoolSize: 150, // Start with 100-200; adjust based on testing
  minPoolSize: 10,
  connectTimeoutMS: 30000,
});

// Connect once on app startup
async function connectDB() {
  try {
    await client.connect();
    console.log('Connected to MongoDB successfully');
  } catch (err) {
    console.error('MongoDB connection failed:', err);
    process.exit(1);
  }
}
connectDB();

module.exports = client;

Update File_Recipes to use this shared client:

const client = require('./db');
const adapt = require("./File_AdaptIngredientOne");

module.exports = async function (shop1, shop2) {
  const db = client.db("database");
  const collection = db.collection("Recipes");
  const collection2 = db.collection("adaptedRecipes");
  
  // Use async/await for cursor iteration instead of callback-based forEach
  const cursor = collection.find(query);
  for await (const doc of cursor) {
    // Wait for async adapters to finish to avoid overwhelming the system
    await Promise.all([
      adaptIngredientOne(doc, collection, collection2, shop1, shop2),
      adaptIngredientTwo(doc, collection, collection2, shop1, shop2)
    ]);
  }
};

Step 2: Control concurrency #

Your nested loops spawn hundreds of recipes() calls at once. Instead of running all combos in parallel, limit concurrent operations to match your CPU thread count (16 for your Ryzen 7 1800). Use a library like p-limit or a simple queue:

const pLimit = require('p-limit');
const limit = pLimit(16); // Match your CPU thread count
const recipes = require("./File_Recipes");
const grocery_stores = ["Aldi", "Lidl", "Edeka", ....];

// Wrap each combo call in the concurrency limit
const tasks = [];
for (let i=0; i < grocery_stores.length; i++) {
  for(let n=i; n < grocery_stores.length; n++) {
    const shop1 = grocery_stores[i];
    const shop2 = grocery_stores[n];
    tasks.push(limit(() => recipes(shop1, shop2)));
  }
}

// Run all tasks with controlled concurrency
await Promise.all(tasks);

Step 3: Optimize MongoDB configuration #

• Tweak connection pool settings: Adjust maxPoolSize in your MongoClient options—start with 100-200 (your 32GB RAM can handle this easily). Avoid setting it too high (e.g., 1000) as it wastes resources on idle connections.
• Add critical indexes: The query collection.findOne({ searchTerm: doc.Name + shop1 }) will be slow without an index. Create one on the searchTerm field (run once via code or MongoDB shell):

  // Run once to create the index
  db.collection.createIndex({ searchTerm: 1 });
  

• Batch inserts: Instead of calling insertOne for every adapted recipe, collect batches of documents and use insertMany to reduce round-trips to MongoDB. For example, in File_AdaptIngredientOne, accumulate documents and insert in batches of 100-500.

Step 4: Tune system limits #

If you still hit connection limits, increase your OS file handle limit:

• On Linux, run ulimit -n 4096 temporarily (or update /etc/security/limits.conf for permanent changes) to allow more open sockets.

Step 5: Fix async/await usage #

Your current File_AdaptIngredientOne is async, but you’re calling it without await in forEach, leading to unmanaged concurrency. The for await...of loop suggested earlier fixes this by waiting for each batch of adapters to finish before moving to the next recipe.

内容的提问来源于stack exchange，提问作者betzebube1000