Flask+uWSGI+NGINX部署后Worker崩溃，遇hr_instance_read错误求助

Hey there, let's break down why your uWSGI workers are crashing after handling a batch of requests, paired with that hr_instance_read(): Connection reset by peer error. First, note that the "connection reset" message often happens when Nginx closes a connection before uWSGI finishes sending a response—but if this is triggering worker crashes, there's usually an underlying issue we need to fix.

Here are the most likely causes and actionable fixes:

1. Worker Resource Exhaustion (Memory/Handle Leaks) #

Your Flask app might have a memory leak that builds up over requests, eventually causing the worker to get killed by the system's OOM (Out of Memory) killer, or forcing uWSGI to terminate it. The connection reset error is just a side effect of the worker crashing mid-request.

• How to check:
  • Use top or htop to monitor worker process memory usage—watch if it climbs steadily until the crash.
  • Check system logs (like /var/log/syslog or run dmesg) for keywords like Out of memory or killed process to confirm OOM kills.
• Fixes:
  • Hunt down memory leaks in your Flask app: Look for global variables that accumulate data, or third-party libraries that don't release resources. Tools like memory_profiler can help pinpoint leaky functions.
  • Configure uWSGI to auto-restart workers before they hit critical limits:
    Add these lines to your uWSGI config:

    max-requests = 1000  # Restart worker after 1000 requests (adjust based on your load)
    reload-on-rss = 200  # Restart if worker uses 200MB RAM (tune to your server's capacity)
    

2. Mismatched Timeout Configs Between Nginx and uWSGI #

If Nginx's timeout settings are shorter than uWSGI's, or uWSGI's harakiri (request timeout kill switch) is too aggressive, it can cause premature connection closures that trigger worker instability.

• How to check:
  • In your Nginx config, verify proxy_connect_timeout, proxy_read_timeout, and proxy_send_timeout—these should be set to a reasonable value (e.g., 30s) instead of being too low.
  • Check uWSGI's harakiri setting: If it's set to something like 5s, workers might get killed mid-processing for slow-but-valid requests.
• Fixes:
  • Sync timeout values across both services. For example:
    Nginx config snippet:

    location / {
        proxy_pass http://uwsgi_backend;
        proxy_connect_timeout 30s;
        proxy_read_timeout 30s;
        proxy_send_timeout 30s;
    }
    

    uWSGI config additions:

    harakiri = 30
    harakiri-verbose = true  # Log when harakiri triggers
    http-timeout = 30
    

3. Uncaught Exceptions in Your Flask App #

A hidden unhandled exception in your code could be taking down workers, with the connection reset error being a secondary symptom. Even if most requests are fast, a specific edge-case request might trigger a crash.

• How to check:
  • Enable detailed logging in Flask:

    import logging
    app = Flask(__name__)
    app.logger.setLevel(logging.DEBUG)
    

    This will log full tracebacks for any uncaught exceptions.
  • Scan your uWSGI logs for Python tracebacks—they'll usually appear right before the connection reset messages if an exception is the root cause.
• Fixes:
  • Add try-except blocks around risky code (like database calls, external API requests, or file operations) to catch and handle exceptions gracefully.
  • Use a global error handler to catch all unhandled exceptions:

    @app.errorhandler(Exception)
    def handle_uncaught_exception(e):
        app.logger.error("Unhandled exception", exc_info=e)
        return "Internal Server Error", 500
    

4. uWSGI Version Bugs #

Older versions of uWSGI have had known bugs with HTTP connection handling that can cause worker crashes.

• How to check:
  • Run uwsgi --version to see your current version, then compare it to the latest stable release.
• Fixes:
  • Upgrade uWSGI to the latest stable version:

    pip install --upgrade uwsgi
    

    If you installed uWSGI via your system package manager (e.g., apt), use apt update && apt upgrade uwsgi instead.

5. Insufficient System File Handle Limits #

If your system has a low limit on open file handles, uWSGI can run out of resources to manage connections, leading to crashes.

• How to check:
  • Run ulimit -n to see the current limit—uWSGI recommends at least 1024, but 4096 is safer for production.
• Fixes:
  • Temporarily raise the limit for the current session:

    ulimit -n 4096
    

  • To make it permanent, edit /etc/security/limits.conf and add:

    * soft nofile 4096
    * hard nofile 4096
    

    Then restart uWSGI and Nginx for changes to take effect.

Start with checking for OOM kills and memory leaks—those are the most common culprits in scenarios like yours. If that doesn't resolve it, move through the other steps one by one.

内容的提问来源于stack exchange，提问作者Vahid Kharazi