为何带负载均衡的WebSocket在高流量下仍随机断开连接？

问题背景 #

基于Node.js + Socket.IO搭建的WebSocket服务器，低负载时运行正常，但当并发客户端数量突破10000+时，出现随机断连、消息延迟、服务器崩溃等问题，使用NGINX作为反向代理。

现有配置 #

NGINX配置 #

server {
    listen 80;
    server_name mydomain.com;

    location /socket.io/ {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
    }
}

Node.js Socket.IO服务器代码 #

const io = require("socket.io")(3000, {
    cors: {
        origin: "*",
    },
});

io.on("connection", (socket) => {
    console.log(`User connected: ${socket.id}`);

    socket.on("message", (data) => {
        console.log(`Received: ${data}`);
        socket.emit("response", "Message received!");
    });

    socket.on("disconnect", () => {
        console.log(`User disconnected: ${socket.id}`);
    });
});

出现的问题 #

• 客户端随机断开连接（高负载时尤为明显）
• 部分消息丢失（违背WebSocket可靠性预期）
• 数千连接时CPU使用率飙升至100%
• NGINX日志显示"Connection reset by peer"错误

已尝试的优化 #

• 增大WebSocket ping间隔：略有改善，但断连仍存在
• 优化NGINX WebSocket配置：无明显效果
• 使用Redis Socket.IO适配器扩容：有一定帮助，但随机断连未解决
• 增加Node.js进程内存：无法避免高负载CPU占用
• 尝试WebSocket服务器集群：客户端重连仍超时

针对性优化方案 #

1. 调整NGINX核心参数，解决连接限制与超时 #

默认NGINX的worker进程数、连接数上限不足以支撑10000+并发，补充以下配置：

# 全局配置段
worker_processes auto;
worker_connections 10240;
multi_accept on;

# HTTP配置段
http {
    keepalive_timeout 65;
    keepalive_requests 10000;

    # WebSocket专属超时与缓冲配置
    proxy_connect_timeout 7d;
    proxy_send_timeout 7d;
    proxy_read_timeout 7d;
    proxy_buffer_size 64k;
    proxy_buffers 4 64k;
    proxy_busy_buffers_size 128k;

    # 负载均衡节点（配合Node.js集群）
    upstream socket_nodes {
        least_conn; # 按最少连接数分配请求
        server localhost:3001;
        server localhost:3002;
        server localhost:3003;
        server localhost:3004;
        # 按服务器CPU核数添加对应端口
    }

    server {
        listen 80;
        server_name mydomain.com;

        location /socket.io/ {
            proxy_pass http://socket_nodes;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            # 传递真实客户端IP
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }
}

worker_connections设为10240确保单worker能处理足够连接，proxy_read_timeout设为7d避免NGINX主动断开长连接，缓冲参数减少IO阻塞。

2. Node.js Socket.IO优化，降低CPU占用与断连 #

关闭高频日志输出

当前代码中每个连接、消息都打印日志，10000+并发时日志IO会占用大量CPU，注释或移除冗余日志：

io.on("connection", (socket) => {
    // console.log(`User connected: ${socket.id}`);

    socket.on("message", (data) => {
        // console.log(`Received: ${data}`);
        socket.emit("response", "Message received!");
    });

    socket.on("disconnect", () => {
        // console.log(`User disconnected: ${socket.id}`);
    });
});

调整心跳与传输参数

默认心跳间隔过于频繁，强制使用WebSocket传输避免轮询开销：

const io = require("socket.io")(3000 + cluster.worker.id, {
    cors: {
        origin: "*",
    },
    pingInterval: 30000, // 30秒发一次ping
    pingTimeout: 60000,   // 60秒没收到pong再断开
    transports: ["websocket"], // 禁用HTTP轮询
});

启用Node.js集群模式，利用多核CPU

单Node.js进程只能占用单核，用cluster模块启动多进程绑定不同端口：

const cluster = require('cluster');
const numCPUs = require('os').cpus().length;
const { createServer } = require('http');
const { Server } = require('socket.io');

if (cluster.isPrimary) {
    console.log(`Primary ${process.pid} is running`);
    // 启动与CPU核数一致的工作进程
    for (let i = 0; i < numCPUs; i++) {
        cluster.fork();
    }
    // 自动重启崩溃的进程
    cluster.on('exit', (worker) => {
        console.log(`worker ${worker.process.pid} died, restarting...`);
        cluster.fork();
    });
} else {
    const httpServer = createServer();
    const io = new Server(httpServer, {
        cors: { origin: "*" },
        pingInterval: 30000,
        pingTimeout: 60000,
        transports: ["websocket"]
    });

    io.on("connection", (socket) => {
        socket.on("message", (data) => {
            socket.emit("response", "Message received!");
        });
    });

    // 每个工作进程绑定不同端口
    httpServer.listen(3000 + cluster.worker.id);
    console.log(`Worker ${process.pid} started on port ${3000 + cluster.worker.id}`);
}

3. 系统级优化，调整文件描述符限制 #

Linux默认文件描述符上限为1024，无法支撑10000+连接，需调高限制：

1. 临时生效（当前会话）：

ulimit -n 65535

2. 永久生效：
   编辑/etc/security/limits.conf：

* soft nofile 65535
* hard nofile 65535
root soft nofile 65535
root hard nofile 65535

编辑/etc/sysctl.conf：

fs.file-max = 65535
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

执行sysctl -p使配置生效。

4. 消息可靠性优化 #

启用Socket.IO的确认机制，确保消息送达：

// 客户端发送消息时等待服务器确认
socket.emit("message", data, (ack) => {
    console.log(ack); // 收到服务器返回的确认信息
});

// 服务器端处理消息并返回确认
socket.on("message", (data, callback) => {
    callback("Message received successfully");
    socket.emit("response", "Message received!");
});

内容的提问来源于stack exchange，提问作者Milan Gohel