AWS EC2 Node.js应用经ELB连Mongo实例遇随机504错误求助

Hey Kris, let's walk through troubleshooting that random 504 Gateway Timeout issue you're hitting when both Node.js instances are behind the ELB. Here's a structured approach to narrow down the root cause:

1. Validate ELB Timeout Configuration #

504 errors often stem from the ELB cutting off the connection before your backend can respond. Check your ELB's Idle Timeout setting (default is 60 seconds):

• If your Node.js app relies on long-running MongoDB queries or connections that stay idle longer than the timeout, the ELB will terminate the connection prematurely.
• Adjust the timeout to match your app's typical request processing time (e.g., if you have queries that take up to 90 seconds, set the ELB timeout to 100 seconds). Also ensure your Node.js app's internal request timeout is slightly shorter than the ELB timeout to avoid the ELB throwing 504s instead of your app returning a proper error.

2. Audit MongoDB Connection Pool Settings #

When two frontend instances connect to MongoDB, you might be hitting connection pool limits, causing requests to wait for available connections and time out.

• Check your Node.js MongoClient connection options: by default, the driver uses a poolSize of 5. If both instances are using the default, that's only 10 concurrent connections to MongoDB. Adjust poolSize based on your app's concurrent request volume (e.g., set to 20 if you expect high traffic).
• Monitor MongoDB's current connection count with this command:

  mongo --host <MONGO_IP> --eval "printjson(db.serverStatus().connections)"
  

  Look for current vs available counts—if available is consistently low or zero, you need to either increase the pool size or raise MongoDB's maxConnections setting (default is 65536, but might be limited by OS file descriptors).

3. Check MongoDB Logs for Connection/Performance Issues #

Dig into MongoDB's logs to spot hidden issues:

• On your MongoDB EC2 instance, check the default log path (usually /var/log/mongodb/mongod.log) for entries like:
  • Too many connections: Indicates you've hit MongoDB's connection limit.
  • Connection refused or SocketException: Points to intermittent network issues across AZs.
  • Slow query logs (if enabled): Look for queries taking longer than expected—these can back up requests when traffic scales to two instances. Enable slow query logging by adding slowms: 100 to your mongod.conf and restarting the service.

4. Analyze ELB and EC2 Metrics via CloudWatch #

Use CloudWatch metrics to pinpoint where the delay is happening:

• Check ELB metrics:
  • HTTPCode_ELB_5XX_Count: Correlate spikes with when both instances are active.
  • TargetResponseTime: See if response times jump when adding the second instance.
  • UnHealthyHostCount: Ensure neither instance is being marked unhealthy incorrectly (which could cause traffic to overload the healthy one).
• Check EC2 metrics for both Node.js instances:
  • CPU, memory, and network utilization: If one instance is consistently under higher load, your ELB's traffic distribution might be uneven, or that instance has underlying issues.

5. Debug Node.js Application Logs #

Your app's logs will tell you exactly what's failing when the 504 occurs:

• Look for errors like MongoTimeoutError, ConnectionPoolExhaustedError, or unhandled promise rejections related to MongoDB connections.
• Ensure your app is properly handling connection errors—if a MongoDB connection drops, does it retry gracefully, or does it leave requests hanging until the ELB times out?
• Add logging around MongoDB query execution times to identify slow operations that might be triggering timeouts under load.

6. Verify Cross-AZ Network Stability #

Even though you've opened ports, cross-AZ networks can have intermittent latency or packet loss:

• Run continuous ping tests from both Node.js instances to the MongoDB instance:

  ping <MONGO_IP> -t
  

  Look for dropped packets or sudden spikes in latency.
• Use the mongo shell to run repeated test queries from both instances simultaneously to simulate load and check for connection interruptions.

内容的提问来源于stack exchange，提问作者Kris