Socket.io+RESTful API经Ingress路由时REST接口504超时问题排查

问题背景 #

开发了一款基于Socket.io的聊天应用，同一服务部署的数据库用于存储聊天记录，RESTful API负责聊天记录的存储与查询。但客户端调用RESTful API时，总会在服务器返回响应前超时，触发504错误，请求疑似阻塞。错误信息涉及Ingress，怀疑Ingress配置存在问题。

相关代码文件 #

服务器文件 #

import express from 'express';
import mongoose from 'mongoose';
import { createServer } from 'http';
import { Server } from 'socket.io';
import { json } from 'body-parser';
import { addUser, getUser, generateMessage } from '@tlbooktrading/common';

// REST API路由中间件
import { indexChatRouter } from './routes';
import { addChatRouter } from './routes/add';

const app = express();
app.set('trust proxy', true);
app.use(json());
app.use(indexChatRouter);
app.use(addChatRouter);

const server = createServer(app);

const io = new Server(server);
const nsp = io.of('/chat');

let socketid: string;

const connectMongo = async () => {
    if (!process.env.MONGO_URI) {
        throw new Error('MONGO_URI must be defined');
    }
    try {
        await mongoose.connect(process.env.MONGO_URI);
        console.log('Connected to MongoDB');
    } catch (error) {
        console.log(error);
    }
};

connectMongo();

nsp.on('connection', (socket) => {
    ...
});

server.listen(3000, () => {
    console.log(`Listening on 3000!`);
});

Ingress配置文件 #

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    name: ingress-service
    annotations:
        kubernetes.io/ingress.class: nginx
        nginx.ingress.kubernetes.io/use-regex: 'true'
        nginx.org/websocket-services: 'chat-srv'
spec:
    rules:
        - host: booktrading.dev
          http:
              paths:
                  # REST endpoints
                  - path: /api/chats/?(.*)
                    pathType: Prefix
                    backend:
                        service:
                            name: chat-srv
                            port:
                                number: 3000
                  # Namespace for socket.io
                  - path: /chat/?(.*)
                    pathType: Prefix
                    backend:
                        service:
                            name: chat-srv
                            port:
                                number: 3000
                  - path: /socket.io/.*
                    pathType: Prefix
                    backend:
                        service:
                            name: chat-srv
                            port:
                                number: 3000
                  - path: /?(.*)
                    pathType: Prefix
                    backend:
                        service:
                            name: client-srv
                            port:
                                number: 3000

Deployment与Service配置文件 #

apiVersion: apps/v1
kind: Deployment
metadata:
    name: chat-depl
spec:
    replicas: 1
    selector:
        matchLabels:
            app: chat
    template:
        metadata:
            labels:
                app: chat
        spec:
            containers:
                - name: chat
                  image: tommyling79/book-trading-chat
                  env:
                      - name: MONGO_URI
                        value: mongodb://chat-mongo-srv:27017/chat
---
apiVersion: v1
kind: Service
metadata:
    name: chat-srv
spec:
    selector:
        app: chat
    ports:
        - name: chat
          protocol: TCP
          port: 3000
          targetPort: 3000

问题分析 #

1. Ingress路径与WebSocket配置冲突

   • nginx.org/websocket-services注解会对chat-srv的所有连接应用WebSocket长超时配置，导致REST API的短连接请求被强制使用长超时，若后端因数据库未就绪阻塞请求，Ingress最终会触发504超时。
   • 开启正则匹配后，pathType: Prefix的行为不符合预期，可能导致路径匹配混乱，比如REST请求被错误路由到WebSocket处理逻辑。

2. MongoDB连接初始化未阻塞服务启动

   • connectMongo()异步执行但未等待完成就启动服务器，导致REST API在数据库未就绪时处理请求，数据库操作阻塞，最终触发Ingress超时。

修复方案 #

1. 调整Ingress配置，分离REST与WebSocket逻辑 #

移除全局WebSocket服务注解，为WebSocket路径单独配置超时，同时修正路径匹配类型：

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/use-regex: 'true'
    # 为WebSocket路径单独配置升级逻辑与超时
    nginx.ingress.kubernetes.io/configuration-snippet: |
      location ~* ^/chat/ {
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
      }
      location ~* ^/socket.io/ {
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
      }
spec:
  rules:
    - host: booktrading.dev
      http:
        paths:
          # REST接口优先匹配
          - path: /api/chats/?(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: chat-srv
                port:
                  number: 3000
          # Socket.io命名空间
          - path: /chat/?(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: chat-srv
                port:
                  number: 3000
          # Socket.io核心路径
          - path: /socket.io/.*
            pathType: ImplementationSpecific
            backend:
              service:
                name: chat-srv
                port:
                  number: 3000
          # 客户端服务
          - path: /?(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: client-srv
                port:
                  number: 3000

2. 等待MongoDB连接完成后启动服务 #

修改服务器代码，确保数据库连接就绪后再对外提供服务：

import express from 'express';
import mongoose from 'mongoose';
import { createServer } from 'http';
import { Server } from 'socket.io';
import { json } from 'body-parser';
import { addUser, getUser, generateMessage } from '@tlbooktrading/common';

import { indexChatRouter } from './routes';
import { addChatRouter } from './routes/add';

const app = express();
app.set('trust proxy', true);
app.use(json());
app.use(indexChatRouter);
app.use(addChatRouter);

const server = createServer(app);

const io = new Server(server);
const nsp = io.of('/chat');

let socketid: string;

const connectMongo = async () => {
    if (!process.env.MONGO_URI) {
        throw new Error('MONGO_URI must be defined');
    }
    try {
        await mongoose.connect(process.env.MONGO_URI);
        console.log('Connected to MongoDB');
    } catch (error) {
        console.error('MongoDB连接失败:', error);
        process.exit(1); // 连接失败退出，让K8s重启容器
    }
};

// 初始化流程：先连数据库，再启动服务
const startServer = async () => {
    await connectMongo();
    server.listen(3000, () => {
        console.log(`Listening on 3000!`);
    });
};

startServer();

nsp.on('connection', (socket) => {
    ...
});

3. 添加就绪探针，确保容器就绪后对外服务 #

在Deployment中添加HTTP就绪探针，避免K8s将未就绪的容器暴露给Ingress：

apiVersion: apps/v1
kind: Deployment
metadata:
    name: chat-depl
spec:
    replicas: 1
    selector:
        matchLabels:
            app: chat
    template:
        metadata:
            labels:
                app: chat
        spec:
            containers:
                - name: chat
                  image: tommyling79/book-trading-chat
                  env:
                      - name: MONGO_URI
                        value: mongodb://chat-mongo-srv:27017/chat
                  # 添加就绪探针，检查数据库连接状态
                  readinessProbe:
                    httpGet:
                      path: /api/chats/health # 需要新增一个健康检查接口，返回200表示就绪
                      port: 3000
                    initialDelaySeconds: 5
                    periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
    name: chat-srv
spec:
    selector:
        app: chat
    ports:
        - name: chat
          protocol: TCP
          port: 3000
          targetPort: 3000

验证步骤 #

1. 应用修改后的配置：kubectl apply -f <配置文件>
2. 检查Pod状态：kubectl get pods，确保chat-depl的Pod处于Ready状态
3. 测试REST API调用，确认不再出现504超时
4. 验证Socket.io连接正常，聊天功能不受影响

内容的提问来源于stack exchange，提问作者tommy-ling