使用Flask-SocketIO+Flask+Gunicorn+Nginx遇502/400错误排查

最近在Ubuntu 18.04上搭了个用Flask+Flask-SocketIO+eventlet+Gunicorn+Nginx的站点，这套配置是Flask-SocketIO官方文档支持的，但跑起来后尴尬了——Flask的普通路由都正常，就是SocketIO完全没反应，只有少数POST请求能返回200 OK。折腾了好一阵终于找到问题所在，把整个过程和解决方法整理出来，希望能帮到遇到同样坑的朋友。

我的环境与初始配置 #

依赖版本（pip3 freeze） #

click==7.1.2
dnspython==1.16.0
eventlet==0.26.1
Flask==1.1.2
Flask-SocketIO==4.3.1
greenlet==0.4.16
gunicorn==19.7.1
itsdangerous==1.1.0
Jinja2==2.11.2
MarkupSafe==1.1.1
monotonic==1.5
pkg-resources==0.0.0
python-engineio==3.13.1
python-socketio==4.6.0
six==1.15.0
Werkzeug==1.0.1

Nginx配置（/etc/nginx/nginx.conf相关部分） #

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
    worker_connections 65535;
    multi_accept on;
}

http {
    ##
    # Basic Settings
    ##
    charset utf-8;
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    server_tokens off;
    log_not_found off;
    client_max_body_size 16M;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # SSL Settings
    ##
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
    ssl_prefer_server_ciphers on;

    ##
    # Logging Settings
    ##
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##
    gzip on;

    ##
    # Virtual Host Configs
    ##
    server {
        listen 80;
        server_name http://compoundingly.com;

        location / {
            proxy_pass http://0.0.0.0:5000;
            proxy_redirect off;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        location /static {
            alias /home/shorya/compoundingly_website/compoundingly_code/static;
            expires 30d;
        }

        location /socket.io {
            include proxy_params;
            proxy_http_version 1.1;
            proxy_buffering off;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_pass http://0.0.0.0:5000/socket.io;
        }
    }

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

Gunicorn服务配置（/etc/systemd/system/gunicorn.service） #

[Unit]
Description=Gunicorn instance to serve myproject
Requires=gunicorn.socket
After=network.target

[Service]
User=root
Group=www-data
WorkingDirectory=/home/shorya/compoundingly_website/compoundingly_code
Environment="PATH=/home/shorya/compoundingly_website/compoundingly_code/compoundingly_venv/bin"
ExecStart=/home/shorya/compoundingly_website/compoundingly_code/compoundingly_venv/bin/gunicorn -k eventlet --workers 3 --bind unix:compoundingly_code.sock -m 007 wsgi:app
Restart=on-failure

[Install]
WantedBy=multi-user.target

初始wsgi.py代码 #

from app import app
if __name__ == "__main__":
    app.run()

错误现象 #

• 部分SocketIO请求返回400 BAD REQUEST：http://compoundingly.com/socket.io/?EIO=3&transport=polling&t=NFGwVcP&sid=22ad9ddeaffe4eeb8525927aca86806b
• 部分SocketIO请求返回502 Bad Gateway：http://compoundingly.com/socket.io/?EIO=3&transport=polling&t=NFGwdji&sid=69f16b45a1aa474294cdeb2452f0e71a
• 仅少数请求返回200 OK：http://compoundingly.com/socket.io/?EIO=3&transport=polling&t=NFGwVtd

Gunicorn错误日志 #

Aug 08 22:09:16 sm01 gunicorn[1956]: [2020-08-08 22:09:16 -0400] [5964] [INFO] Booting worker with pid: 5964
Aug 08 22:09:47 sm01 gunicorn[1956]: [2020-08-08 22:09:47 -0400] [1956] [CRITICAL] WORKER TIMEOUT (pid:5962)
Aug 08 22:09:47 sm01 gunicorn[1956]: [2020-08-08 22:09:47 -0400] [5962] [INFO] Worker exiting (pid: 5962)
Aug 08 22:09:47 sm01 gunicorn[1956]: [2020-08-08 22:09:47 -0400] [5977] [INFO] Booting worker with pid: 5977
Aug 08 22:09:48 sm01 gunicorn[1956]: [2020-08-08 22:09:48 -0400] [1956] [CRITICAL] WORKER TIMEOUT (pid:5963)
Aug 08 22:09:48 sm01 gunicorn[1956]: [2020-08-08 22:09:48 -0400] [1956] [CRITICAL] WORKER TIMEOUT (pid:5964)
Aug 08 22:09:48 sm01 gunicorn[1956]: [2020-08-08 22:09:48 -0400] [5963] [INFO] Worker exiting (pid: 5963)
Aug 08 22:09:48 sm01 gunicorn[1956]: [2020-08-08 22:09:48 -0400] [5964] [INFO] Worker exiting (pid: 5964)
Aug 08 22:09:48 sm01 gunicorn[1956]: [2020-08-08 22:09:48 -0400] [5981] [INFO] Booting worker with pid: 5981
Aug 08 22:09:48 sm01 gunicorn[1956]: [2020-08-08 22:09:48 -0400] [5982] [INFO] Booting worker with pid: 5982

解决方法 #

经过反复测试，以下三个步骤结合起来解决了问题：

1. 修改wsgi.py，确保SocketIO实例正确暴露 #

原来的wsgi只导出了Flask的app，但SocketIO需要自己的实例来处理连接。修改后的代码：

from app import *
if __name__ == "__main__":
    # app.run()
    app = Flask(__name__)
    app.config['SECRET_KEY'] = 'secret!'
    socketio = SocketIO(app, async_mode="eventlet")
    socketio.run(app, host="0.0.0.0", port=5000, debug=True)

注：如果你的SocketIO实例是在app.py里定义的，也可以直接从app导入socketio，然后让Gunicorn运行socketio的wsgi_app属性，比如在wsgi.py里写from app import socketio; app = socketio.wsgi_app，这样更简洁。

2. 调整Gunicorn的绑定参数 #

原来的Gunicorn绑定的是Unix socket，但结合eventlet使用时，直接绑定到0.0.0.0:5000更稳定。修改Gunicorn服务配置里的ExecStart行：

ExecStart=/home/shorya/compoundingly_website/compoundingly_code/compoundingly_venv/bin/gunicorn -k eventlet --workers 3 --bind 0.0.0.0:5000 -m 007 wsgi:app

3. 更换Gunicorn版本 #

测试发现Gunicorn 19.7.1和20.0.4都存在兼容性问题，换成20.0.3就正常了：

pip install gunicorn==20.0.3

做完这些修改后，重启Nginx和Gunicorn服务：

sudo systemctl restart nginx
sudo systemctl restart gunicorn

再测试SocketIO连接，就完全正常了！

内容的提问来源于stack exchange，提问作者smal