重启Redis后HAProxy无法检测服务，请求排查配置问题

环境信息 #

当前Redis集群相关K8s资源状态：

NAME                                            READY   STATUS    RESTARTS   AGE
pod/redis-haproxy-deployment-65497cd78d-659tq   1/1     Running   0          31m
pod/redis-sentinel-node-0                       3/3     Running   0          81m
pod/redis-sentinel-node-1                       3/3     Running   0          80m
pod/redis-sentinel-node-2                       3/3     Running   0          80m
pod/ubuntu                                      1/1     Running   0          85m

NAME                              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)              AGE
service/redis-haproxy-balancer    ClusterIP   10.43.92.106   <none>        6379/TCP             31m
service/redis-sentinel-headless   ClusterIP   None           <none>        6379/TCP,26379/TCP   99m
service/redis-sentinel-metrics    ClusterIP   10.43.72.97    <none>        9121/TCP             99m

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/redis-haproxy-deployment   1/1     1            1           31m

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/redis-haproxy-deployment-65497cd78d   1         1         1       31m

NAME                                   READY   AGE
statefulset.apps/redis-sentinel-node   3/3     99m

之前正常的连接方式 #

原本可通过HAProxy服务连接主Redis：

redis-cli -h redis-haproxy-balancer

redis-haproxy-balancer:6379> keys *
1) "sdf"
2) "sdf12"
3) "s4df12"
4) "s4df1"
5) "fsafsdf"
6) "!s4d!1"
7) "s4d!1"

HAProxy及K8s相关配置 #

haproxy.cfg #

global
  daemon
  maxconn 256


defaults REDIS
  mode tcp
  timeout connect 3s
  timeout server 3s
  timeout client 3s


frontend front_redis
  bind 0.0.0.0:6379
  use_backend redis_cluster


backend redis_cluster
  mode tcp
  option tcp-check
  tcp-check comment PING\ phase
  tcp-check send PING\r\n
  tcp-check expect string +PONG
  tcp-check comment role\ check
  tcp-check send info\ replication\r\n
  tcp-check expect string role:master
  tcp-check comment QUIT\ phase
  tcp-check send QUIT\r\n
  tcp-check expect string +OK     

  server redis-0 redis-sentinel-node-0.redis-sentinel-headless:6379 maxconn 1024 check inter 1s 
  server redis-1 redis-sentinel-node-1.redis-sentinel-headless:6379 maxconn 1024 check inter 1s 
  server redis-2 redis-sentinel-node-2.redis-sentinel-headless:6379 maxconn 1024 check inter 1s 

haproxy-service.yaml #

apiVersion: v1 
kind: Service 
metadata: 
  name: redis-haproxy-balancer
spec: 
  type: ClusterIP
  selector: 
    app: redis-haproxy 
  ports: 
  - protocol: TCP
    port: 6379 
    targetPort: 6379

redis-haproxy-deployment.yaml #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-haproxy-deployment
  labels:
    app: redis-haproxy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-haproxy
  template:
    metadata:
      labels:
        app: redis-haproxy
    spec:
      containers:
      - name: redis-haproxy
        image: haproxy:lts-alpine
        volumeMounts:
        - name: redis-haproxy-config-volume
          mountPath: /usr/local/etc/haproxy/haproxy.cfg
          subPath: haproxy.cfg 
        ports:
        - containerPort: 6379
      volumes:
      - name: redis-haproxy-config-volume
        configMap:
          name: redis-haproxy-config
          items:
          - key: haproxy.cfg
            path: haproxy.cfg  

问题现象 #

Redis重启后，无法通过redis-haproxy-balancer连接服务，HAProxy日志显示：

[NOTICE]   (1) : New worker (8) forked
[NOTICE]   (1) : Loading success.
[WARNING]  (8) : Server redis_cluster/redis-0 is DOWN, reason: Layer7 timeout, info: " at step 6 of tcp-check (expect string 'role:master')", check duration: 1000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (8) : Server redis_cluster/redis-1 is DOWN, reason: Layer7 timeout, info: " at step 6 of tcp-check (expect string 'role:master')", check duration: 1005ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (8) : Server redis_cluster/redis-2 is DOWN, reason: Layer7 timeout, info: " at step 6 of tcp-check (expect string 'role:master')", check duration: 1001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (8) : backend 'redis_cluster' has no server available!

直接连接redis-sentinel-node-0.redis-sentinel-headless可正常访问。

问题原因及解决方案 #

核心原因 #

日志显示所有节点在检查role:master步骤超时，问题出在两个方面：

1. 超时时间设置过短：当前defaults REDIS里timeout server仅3秒，且未单独设置TCP检查超时，Redis重启后可能处于负载较高、响应延迟的状态，导致info replication命令的响应超过HAProxy的检查超时阈值。
2. Redis主节点切换：重启后集群可能触发主从切换，原主节点变为从节点，HAProxy会将其标记为down；而新主节点的检查因超时未通过，最终导致后端无可用服务器。

修复方案 #

1. 调整超时配置

修改haproxy.cfg，延长超时时间并给TCP检查单独设置超时：

defaults REDIS
  mode tcp
  timeout connect 10s  # 延长连接超时
  timeout server 10s   # 延长服务器响应超时
  timeout client 10s   # 延长客户端超时

backend redis_cluster
  mode tcp
  option tcp-check
  tcp-check timeout 5s  # 单独设置TCP检查的超时时间，避免受全局server timeout影响
  tcp-check comment PING\ phase
  tcp-check send PING\r\n
  tcp-check expect string +PONG
  tcp-check comment role\ check
  tcp-check send info\ replication\r\n
  tcp-check expect string role:master
  tcp-check comment QUIT\ phase
  tcp-check send QUIT\r\n
  tcp-check expect string +OK     

2. 优化检查逻辑（可选）

如果Redis开启了密码认证，需要在TCP检查前添加AUTH命令（当前直接连接正常，说明无密码或已处理）；另外可调整检查间隔，比如将check inter 1s改为check inter 3s，降低检查频率，减少Redis负载。

3. 应用配置更新

修改ConfigMap后，重启HAProxy Deployment使配置生效：

kubectl rollout restart deployment redis-haproxy-deployment

内容的提问来源于stack exchange，提问作者cevoye