Google Cloud虚拟机CPU占用过高（均值160%）排查求助

Hey there, let’s walk through how to figure out why your Node.js app on GCP’s g1-small instance is hitting such high CPU usage (up to 175%, average 160%). Even with non-blocking code, there are plenty of hidden culprits—here’s a practical, step-by-step troubleshooting plan:

Before diving into code, rule out system-level or non-app causes:

• Use GCP Monitoring first: Check the CPU breakdown in GCP Cloud Monitoring to see if the Node.js process is the main offender, or if other system processes (like kswapd for swap, or logging services) are contributing.
• Log into the instance for real-time checks:
  • Run htop to visualize CPU usage per process/thread—Node.js runs a main thread but uses a libuv thread pool, so keep an eye on the main Node process’s CPU percentage.
  • Use pidstat -p <YOUR_NODE_PID> 1 to track the Node process’s CPU usage over time (get the PID with ps aux | grep node).

Non-blocking code doesn’t guarantee low CPU—here’s what to look for:

• Event loop blocking from long synchronous tasks: Even non-blocking apps can have accidental sync code that chokes the event loop. Examples include:
  • Complex calculations (like large data transformations) done synchronously
  • Synchronous file operations (fs.readFileSync instead of fs.readFile)
  • Tight loops without yielding to the event loop (e.g., while (true) without setImmediate or process.nextTick)
  • How to diagnose: Use Chrome DevTools’ Performance panel (launch your app with node --inspect, then connect via chrome://inspect) to record a session—look for "Long Tasks" that block the main thread for 50ms+. Alternatively, use clinic bubbleprof to map event loop delays.
• Infinite loops or runaway code: A classic culprit—double-check for loops that don’t terminate, or recursive functions without proper base cases. Even a loop processing millions of items synchronously can peg the CPU.
• GC thrashing from memory leaks: If your app is leaking memory, Node.js will run garbage collection (GC) constantly. GC is a synchronous operation that eats CPU.
  • How to diagnose: Take heap snapshots in Chrome DevTools’ Memory panel over time—look for objects that keep growing in count. You can also use node --expose-gc and manually trigger GC in your code to measure how often it runs.
• Troublesome third-party libraries: Sometimes npm dependencies have hidden sync code or inefficient algorithms. Try temporarily disabling non-critical libraries to see if CPU usage drops. Check the npm package’s issues page for known performance bugs.

Your g1-small instance has 1.7GB of memory—memory pressure can indirectly spike CPU:

• Swap usage: If your app is using near 100% of memory, the system will use swap (disk-based memory), which is slow and forces the CPU to handle constant data transfer between RAM and disk. Check swap usage with free -m—if Swap Used is high, memory pressure is likely contributing to CPU usage.
• Shared CPU limits: g1-small uses shared CPU—if other tenants on the same physical host are using resources, you might see CPU throttling or spikes. But GCP’s monitoring should show if this is the case (look for "CPU throttled" metrics).

Google’s recommendation to move to a custom v1 CPU instance with 1.75GB of memory is worth a quick test:

• Deploy your app to this temporary instance and monitor CPU usage. If it drops significantly, the issue is likely tied to memory pressure (the extra 50MB might be enough to avoid swap and reduce GC frequency). But don’t stop here—still investigate the root cause so you don’t just band-aid the problem.

If the above steps don’t find the issue, use Node.js’s built-in tools to pinpoint exact functions:

• Built-in profiler: Launch your app with node --prof app.js, let it run under load for a few minutes, then shut it down. You’ll get a isolate-xxxx-v8.log file—process it with node --prof-process isolate-xxxx-v8.log to see a breakdown of CPU usage by function.
• Clinic.js tools: This official Node.js toolkit simplifies profiling:
  • clinic doctor runs an automated diagnosis of CPU, memory, and event loop issues, then gives actionable recommendations.
  • clinic flame generates a flame map to visualize which function calls are consuming the most CPU.

内容的提问来源于stack exchange，提问作者Quazi