AKS上Keycloak StatefulSet多Pod集群发现失败求助

问题描述 #

在AKS上通过StatefulSet部署Keycloak 21.1.1高可用实例，已配置生产模式、证书，指定无头服务FQDN，但多Pod启动后均作为coordinator运行，无法互相发现。

当前配置 #

StatefulSet配置 #

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: keycloak-statefulset
spec:
  serviceName: keycloak-svc
  replicas: 3
  selector:
    matchLabels:
      app: keycloak
  template:
    metadata:
      labels:
        app: keycloak
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "8443"
        prometheus.io/scrape: "true"
        prometheus.io/scheme: https
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: keycloak
                topologyKey: kubernetes.io/zone
              weight: 1
      serviceAccountName: keycloak-svc-account
      containers:
      - env:
          - name: DB_HOST
            value: "10.0.194.2"
          - name: DB_PORT
            value: "5432"
          - name: DB_USERNAME
            value: keycloak
          - name: DB_PASSWORD
            valueFrom:
              secretKeyRef:
                name: keycloak-postgres-secret
                key: keycloak-password
          - name: KEYCLOAK_ADMIN
            value: "admin"
          - name: KEYCLOAK_ADMIN_PASSWORD
            valueFrom:
              secretKeyRef:
                name: keycloak-secret
                key: password
          - name: KC_CACHE
            value: "ispn"
          - name: JAVA_OPTS
            value: -Djgroups.dns.query=keycloak-svc.amq-keycloak-dev.svc.cluster.local
          - name: KC_CACHE_STACK
            value: kubernetes
        name: keycloak
        image: "harbor.net/keycloak:main" # 非真实镜像地址
        imagePullPolicy: Always
        resources:
          requests:
            memory: "1G"
            cpu: "250m"
          limits:
            memory: "1G"
            cpu: "500m"
        ports:
          - containerPort: 8443
            name: console
            protocol: TCP
        volumeMounts:
        - name: security-keystore-config
          mountPath: /vault/keystore
        - name: security-truststore-config
          mountPath: /vault/truststore
      volumes:
        - name: security-truststore-config
          secret:
            secretName: truststore-cert-keycloak
        - name: security-keystore-config
          secret:
            secretName: keystore-cert-keycloak

Dockerfile #

SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN apt-get -qq -o=Dpkg::Use-Pty=0 update && apt-get -y install wget
RUN apt-get -y install libaio1
RUN rm -rf /var/lib/apt/lists/*

COPY keycloak-event-metrics-1.0.1-SNAPSHOT.jar /opt/keycloak-event-metrics-1.0.1-SNAPSHOT.jar

WORKDIR /opt/keycloak
RUN wget https://github.com/keycloak/keycloak/releases/download/21.1.1/keycloak-21.1.1.tar.gz

ENV KC_DB=postgres
ENV KC_HOSTNAME_STRICT=false
ENV KC_HEALTH_ENABLED=true 
ENV KC_METRICS_ENABLED=true
ENV KC_METRICS_STATS_ENABLED=true
ENV KC_FEATURES=token-exchange
ENV KC_PROXY=passthrough
ENV KC_DB_POOL_INITIAL_SIZE=10
ENV KC_DB_POOL_MIN_SIZE=10
ENV KC_DB_POOL_MAX_SIZE=25
ENV KC_LOG_CONSOLE_COLOR=true

EXPOSE 8443

COPY docker-entrypoint.sh /assets/docker-entrypoint.sh
ENTRYPOINT ["/assets/docker-entrypoint.sh"]

entrypoint脚本 #

#!/bin/bash
set -e

tar -xvzf /opt/keycloak/keycloak-21.1.1.tar.gz > /dev/null
rm /opt/keycloak/keycloak-21.1.1.tar.gz
cp /opt/keycloak-event-metrics-1.0.1-SNAPSHOT.jar /opt/keycloak/keycloak-21.1.1/providers/

# 修复：此处多了一个等号，会导致SSL_PASS变量赋值错误
SSL_PASS=$(date +%s | sha256sum | base64 | head -c 32 ; echo)
folder_path="/vault/truststore"
cer_files=($(find "$folder_path" -type f -name "*.cer"))

mkdir -p /vault/certificate
truststore_path="/vault/certificate/truststore.jks"

for cer_file in "${cer_files[@]}"; do
    filename=$(basename "$cer_file")
    alias_name="${filename%.*}"
    keytool -importcert -alias "$alias_name" -file "$cer_file" -keystore "$truststore_path" -storepass "$SSL_PASS" -noprompt > /dev/null 2>&1
done

folder_path="/vault/keystore/"
cer_file=($(find "$folder_path" -type f -name "*.cer"))
key_file=($(find "$folder_path" -type f -name "*.key"))
openssl pkcs12 -export -out /vault/certificate/keystore.p12 -inkey "$key_file" -in "$cer_file" -passout pass:"$SSL_PASS" > /dev/null 2>&1

keytool -importkeystore -srckeystore /vault/certificate/keystore.p12 -srcstoretype PKCS12 -destkeystore /vault/certificate/server.keystore -deststoretype JKS -srcstorepass "$SSL_PASS" -deststorepass "$SSL_PASS" > /dev/null 2>&1

export KC_DB_URL="jdbc:postgresql://$DB_HOST:$DB_PORT/keycloak?useSSL=false"
export KC_SPI_TRUSTSTORE_FILE_PASSWORD=$SSL_PASS
export KC_SPI_TRUSTSTORE_FILE_FILE=/vault/certificate/truststore.jks
export KC_HTTPS_KEY_STORE_FILE=/vault/certificate/server.keystore
export KC_HTTPS_KEY_STORE_PASSWORD=$SSL_PASS
export KC_DB_USERNAME=$DB_USERNAME
export KC_DB_PASSWORD=$DB_PASSWORD


/opt/keycloak/keycloak*/bin/kc.sh build
/opt/keycloak/keycloak*/bin/kc.sh start

解决方案 #

1. 修复entrypoint变量赋值错误 #

entrypoint脚本中SSL_PASS==$(...)存在语法错误，多了一个等号会导致证书密码变量失效，进而影响集群通信的SSL配置。修正为：

SSL_PASS=$(date +%s | sha256sum | base64 | head -c 32 ; echo)

2. 完善集群发现环境变量 #

Keycloak 21.1.1使用Infinispan作为缓存，结合Kubernetes集群发现，需补充以下环境变量（或更新JAVA_OPTS）：

- name: KC_JGROUPS_DISCOVERY_PROTOCOL
  value: dns.DNS_PING
- name: KC_JGROUPS_DISCOVERY_PROPERTIES
  value: "dns_query=keycloak-svc.amq-keycloak-dev.svc.cluster.local"
- name: KC_JGROUPS_TRANSPORT_STACK
  value: tcp

或更新JAVA_OPTS确保JGroups能正确解析无头服务下的所有Pod：

- name: JAVA_OPTS
  value: >-
    -Djgroups.dns.query=keycloak-svc.amq-keycloak-dev.svc.cluster.local
    -Djgroups.transport.type=tcp
    -Djgroups.bind.port=7600

3. 暴露JGroups通信端口 #

JGroups默认使用7600端口进行集群节点通信，需在StatefulSet容器端口配置中添加该端口：

ports:
  - containerPort: 8443
    name: console
    protocol: TCP
  - containerPort: 7600
    name: jgroups
    protocol: TCP

4. 确认无头服务配置正确 #

确保keycloak-svc是ClusterIP为None的无头服务，且标签选择器与StatefulSet的Pod标签匹配，示例配置：

apiVersion: v1
kind: Service
metadata:
  name: keycloak-svc
spec:
  clusterIP: None
  selector:
    app: keycloak
  ports:
    - port: 8443
      name: console
    - port: 7600
      name: jgroups

5. 优化启动命令 #

在entrypoint脚本中使用生产模式优化启动命令，确保集群配置生效：

/opt/keycloak/keycloak*/bin/kc.sh start --optimized

6. 验证集群状态 #

Pod启动后，通过以下命令检查集群状态：

# 查看缓存配置
kubectl exec -it keycloak-statefulset-0 -- /opt/keycloak/keycloak-21.1.1/bin/kc.sh show-config | grep cache
# 查看JGroups日志
kubectl logs keycloak-statefulset-0 | grep JGRP

内容的提问来源于stack exchange，提问作者Subhidh Agarwal