Kubernetes集群中同时运行两组Redis关联部署时出现间歇性连接失败

环境与问题概述 #

• 集群内管理四个部署：
  • Laravel队列工作负载：标签laravel-kube-worker
  • Django队列工作负载：标签django-kube-worker
  • Laravel Redis实例：标签laravel-kube-worker-redis
  • Django Redis实例：标签django-kube-worker-redis
• 预期行为：Laravel工作负载Pod连接对应Redis处理队列任务，Django工作负载Pod同理连接自身对应Redis
• 异常表现：单独运行Laravel或Django部署时一切正常，队列可正常填充与处理；同时运行两组部署时，工作负载Pod会间歇性无法连接到各自的Redis Pod，连接状态不稳定（约15秒内时断时续）
• 已排查情况：两组Redis配置无冲突，Pod、Service、StatefulSet状态均正常，无Redis Pod强制重启记录

相关配置文件 #

Laravel Redis Service配置（括号内为Django Redis对应配置） #

apiVersion: v1
kind: Service
metadata:
  name: laravel-kube-worker-redis  (( django-kube-worker-redis ))
  labels:
    tier: backend
    layer: redis
spec:
  ports:
    - port: 6379  (( 6380 ))
      targetPort: 6379  (( 6380 ))
      nodePort: 32379  (( 32380 ))
      protocol: TCP
  selector:
    tier: backend
    layer: redis
  type: NodePort

Laravel Redis持久化卷声明（括号内为Django Redis对应配置） #

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: laravel-kube-worker-redis  (( django-kube-worker-redis ))
spec:
  storageClassName: do-block-storage 
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Laravel Redis有状态集（括号内为Django Redis对应配置） #

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: laravel-kube-worker-redis  (( django-kube-worker-redis ))
  labels:
    tier: backend
    layer: redis
spec:
  serviceName: laravel-kube-worker-redis  (( django-kube-worker-redis ))
  selector:
    matchLabels:
      tier: backend
      layer: redis
  replicas: 1
  template:
    metadata:
      labels:
        tier: backend
        layer: redis
    spec:
      containers:
        - name: redis
          image: redis:alpine
          command: ["redis-server", "--appendonly", "yes", "--bind", "0.0.0.0"]
          ports:
            - containerPort: 6379  (( 6380 ))
              name: web
          volumeMounts:
            - name: redis-aof
              mountPath: /data
          resources:
            requests:
              memory: "1048Mi" # Set the minimum memory request
            limits:
              memory: "4096Mi" # Set the maximum memory limit
      volumes:
        - name: redis-aof
          persistentVolumeClaim:
            claimName: laravel-kube-worker-redis  (( django-kube-worker-redis ))

关键说明 #

• Django Redis的所有标签键值与Laravel Redis完全不同
• Laravel工作负载Pod通过laravel-kube-worker-redis:6379连接对应Redis环境，Django工作负载Pod通过django-kube-worker-redis:6380连接自身Redis环境

内容的提问来源于stack exchange，提问作者Delano van der Waal