单节点Kubernetes中3副本RabbitMQ Pod流量处理及队列异常问题

问题描述 #

在单节点Kubernetes环境中部署了3个副本的RabbitMQ，出现队列无法正常同步的问题，目前通过Service强制将所有流量转发至rabbitmq-0 Pod，但这种方式存在弊端：RabbitMQ版本更新时会导致服务中断，且应用如果意外连接到其他节点会出现消息无法正常接收的情况。

当前使用的rabbitmq.yml配置如下：

---
apiVersion: v1
kind: Namespace
metadata:
  name: rabbitmq-test
  labels:
    name: rabbitmq-test
---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
  namespace: rabbitmq-test
  labels:
    app: rabbitmq
spec:
  type: NodePort
  ports:
    - name: amqp
      nodePort: 30000
      port: 5672
      protocol: TCP
      targetPort: 5672
    - name: management
      nodePort: 30001
      port: 15672
      protocol: TCP
      targetPort: 15672
  selector:
    app: rabbitmq
    statefulset.kubernetes.io/pod-name: rabbitmq-0
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: rabbitmq-test
spec:
  selector:
    matchLabels:
      app: rabbitmq
  serviceName: "rabbitmq"
  minReadySeconds: 10
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      volumes:
        - name: rabbitmq-storage
          persistentVolumeClaim:
            claimName: rabbitmq-pvc
      terminationGracePeriodSeconds: 10
      containers:
        - name: rabbitmq
          image: rabbitmq:3.11.3-management
          lifecycle:
            postStart:
              exec:
                command: ["/bin/sh", "-c", "cp /mnt/data/test/rabbitmq_delayed_message_exchange-3.11.1.ez /opt/rabbitmq/plugins/ && rabbitmq-plugins --offline enable rabbitmq_peer_discovery_k8s rabbitmq_delayed_message_exchange"]
          imagePullPolicy: Always
          env:
            - name: RABBITMQ_DEFAULT_USER
              value: ""
            - name: RABBITMQ_DEFAULT_PASS
              value: ""
            - name: RABBITMQ_DEFAULT_VHOST
              value: ""
          ports:
            - name: amqp
              containerPort: 5672
            - name: management
              containerPort: 15672
          volumeMounts:
            - mountPath: "/mnt/data/test"
              name: rabbitmq-storage
          resources:
            requests:
              cpu: 500m
              memory: 256Mi
            limits:
              cpu: 1000m
              memory: 512Mi
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: rabbitmq-hpa
  namespace: rabbitmq-test
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: rabbitmq
  minReplicas: 3
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 60
  behavior:
    scaleDown:
      policies:
        - type: Pods
          value: 2
          periodSeconds: 60
        - type: Percent
          value: 5
          periodSeconds: 60

问题根源分析 #

1. Service绑定单一Pod：当前Service的selector指定了statefulset.kubernetes.io/pod-name: rabbitmq-0，所有流量仅流向该Pod，其他节点无法参与集群，队列数据无法同步。
2. 集群配置缺失：虽然启用了rabbitmq_peer_discovery_k8s插件，但缺少Erlang Cookie、K8s集群发现的关键环境变量，节点无法自动组成集群。
3. 共享存储冲突：所有Pod挂载同一个PVC，RabbitMQ集群每个节点需要独立的持久化存储，共享存储会导致节点间数据冲突。
4. 集群通信端口未暴露：RabbitMQ集群需要4369（epmd端口）、25672（集群通信端口），当前配置未包含这些端口，节点间无法通信。
5. HPA配置不合理：RabbitMQ集群扩缩容需要手动处理节点加入/退出逻辑，基于CPU/内存的自动扩缩容会导致集群状态不稳定。

优化后的配置方案 #

以下是修正后的完整配置，解决队列同步问题并移除单一Pod绑定的弊端：

---
apiVersion: v1
kind: Namespace
metadata:
  name: rabbitmq-test
  labels:
    name: rabbitmq-test
---
# Headless Service，用于StatefulSet节点的稳定网络标识
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-headless
  namespace: rabbitmq-test
  labels:
    app: rabbitmq
spec:
  clusterIP: None
  ports:
    - name: amqp
      port: 5672
      targetPort: 5672
    - name: management
      port: 15672
      targetPort: 15672
    - name: epmd
      port: 4369
      targetPort: 4369
    - name: cluster
      port: 25672
      targetPort: 25672
  selector:
    app: rabbitmq
---
# 对外服务的Service，流量分发到所有RabbitMQ节点
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
  namespace: rabbitmq-test
  labels:
    app: rabbitmq
spec:
  type: NodePort
  ports:
    - name: amqp
      nodePort: 30000
      port: 5672
      protocol: TCP
      targetPort: 5672
    - name: management
      nodePort: 30001
      port: 15672
      protocol: TCP
      targetPort: 15672
  selector:
    app: rabbitmq
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: rabbitmq-test
spec:
  selector:
    matchLabels:
      app: rabbitmq
  serviceName: "rabbitmq-headless" # 关联Headless Service
  replicas: 3
  minReadySeconds: 10
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates: # 每个Pod独立PVC
    - metadata:
        name: rabbitmq-storage
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      terminationGracePeriodSeconds: 60 # 延长优雅终止时间，确保集群节点正常退出
      containers:
        - name: rabbitmq
          image: rabbitmq:3.11.3-management
          lifecycle:
            postStart:
              exec:
                command: ["/bin/sh", "-c", "cp /mnt/data/test/rabbitmq_delayed_message_exchange-3.11.1.ez /opt/rabbitmq/plugins/ && rabbitmq-plugins --offline enable rabbitmq_peer_discovery_k8s rabbitmq_delayed_message_exchange"]
          imagePullPolicy: Always
          env:
            - name: RABBITMQ_DEFAULT_USER
              value: "admin" # 替换为实际用户名
            - name: RABBITMQ_DEFAULT_PASS
              value: "admin123" # 替换为实际密码
            - name: RABBITMQ_DEFAULT_VHOST
              value: "/"
            - name: RABBITMQ_ERLANG_COOKIE
              value: "rabbitmq_cluster_cookie" # 所有节点必须使用相同的Cookie
            - name: RABBITMQ_NODENAME
              value: "rabbit@$(hostname).rabbitmq-headless.rabbitmq-test.svc.cluster.local"
            - name: RABBITMQ_USE_LONGNAME
              value: "true"
            - name: RABBITMQ_K8S_SERVICE_NAME
              value: "rabbitmq-headless"
            - name: RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS
              value: "-rabbit cluster_nodes {['rabbit@rabbitmq-0.rabbitmq-headless.rabbitmq-test.svc.cluster.local'],disc}"
          ports:
            - name: amqp
              containerPort: 5672
            - name: management
              containerPort: 15672
            - name: epmd
              containerPort: 4369
            - name: cluster
              containerPort: 25672
          volumeMounts:
            - mountPath: "/var/lib/rabbitmq" # 修改为RabbitMQ默认数据目录
              name: rabbitmq-storage
            - mountPath: "/mnt/data/test" # 挂载插件目录
              name: rabbitmq-storage # 如果插件是存储在PVC中，否则需要单独配置
          resources:
            requests:
              cpu: 500m
              memory: 256Mi
            limits:
              cpu: 1000m
              memory: 512Mi
---
# 建议暂时关闭HPA，RabbitMQ集群扩缩容需手动操作
# apiVersion: autoscaling/v2
# kind: HorizontalPodAutoscaler
# metadata:
#   name: rabbitmq-hpa
#   namespace: rabbitmq-test
# spec:
#   scaleTargetRef:
#     apiVersion: apps/v1
#     kind: StatefulSet
#     name: rabbitmq
#   minReplicas: 3
#   maxReplicas: 5
#   metrics:
#     - type: Resource
#       resource:
#         name: cpu
#         target:
#           type: Utilization
#           averageUtilization: 50
#     - type: Resource
#       resource:
#         name: memory
#         target:
#           type: Utilization
#           averageUtilization: 60
#   behavior:
#     scaleDown:
#       policies:
#         - type: Pods
#           value: 2
#           periodSeconds: 60
#         - type: Percent
#           value: 5
#           periodSeconds: 60

配置说明 #

1. Headless Service：为StatefulSet提供稳定的DNS记录，确保RabbitMQ节点能通过域名互相发现。
2. 独立PVC：通过volumeClaimTemplates为每个Pod创建独立的持久化存储，避免数据冲突。
3. 集群环境变量：
   • RABBITMQ_ERLANG_COOKIE：所有节点共享的Cookie，是集群通信的凭证。
   • RABBITMQ_NODENAME：节点的完整域名，确保集群内节点能正确识别彼此。
   • RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS：指定集群的初始节点，新节点启动后会自动加入集群。
4. 暴露集群端口：添加4369和25672端口，保证节点间集群通信正常。
5. 关闭HPA：RabbitMQ集群自动扩缩容需要额外的集群管理逻辑，建议手动处理节点的增减。

验证步骤 #

1. 部署配置后，等待所有Pod启动完成：

   kubectl get pods -n rabbitmq-test
   

2. 进入任意Pod，查看集群节点状态：

   kubectl exec -it rabbitmq-0 -n rabbitmq-test -- rabbitmqctl cluster_status
   

3. 通过管理界面（NodePort 30001）登录，查看集群节点列表，确认所有节点已加入集群。
4. 创建测试队列，发送消息后，在其他节点查看队列是否同步消息。

内容的提问来源于stack exchange，提问作者someuser