无法通过ScaledObject将KEDA连接至AWS MSK的问题排查

环境与配置概述 #

• AWS MSK集群：3个broker，启用IAM角色认证和SASL/SCRAM认证，微服务已正常通过该集群收发消息
• EKS集群：微服务部署在default命名空间，KEDA安装在keda命名空间
• 认证配置：在default命名空间创建了基于SASL/SCRAM的TriggerAuthentication，关联存储用户名密码的Secret

Secret与TriggerAuthentication配置 #

apiVersion: v1
kind: Secret
metadata:
  name: keda-kafka-secrets
  namespace: {{ .Release.Namespace }}
data:
  username: "{{ $.Values.kafka.username | b64enc }}"
  password: "{{ $.Values.kafka.password | b64enc }}"
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-kafka-conn
  namespace: {{ .Release.Namespace }}
spec:
  secretTargetRef:
  - parameter: username
    name: keda-kafka-secrets
    key: username
  - parameter: password
    name: keda-kafka-secrets
    key: password

ScaledObject配置（微服务Helm Chart中） #

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: user-service-scaledobject
spec:
  scaleTargetRef:
    kind: Deployment
    name: {{ $.Chart.Name }}
  pollingInterval: 30 # Optional. Default: 30 seconds
  cooldownPeriod:  300 # Optional. Default: 300 seconds
  idleReplicaCount: 0 # Optional. Default: 0
  minReplicaCount: 0 # Optional. Default: 0
  maxReplicaCount: 100 # Optional. Default: 100
  triggers:
  - type: kafka
    authenticationRef:
        name: keda-trigger-auth-kafka-conn
    metadata:
        bootstrapServers: {{ .Values.kafka.bootstrapServers }}
        consumerGroup: {{ .Values.kafka.consumerGroup }}
        topic: {{ .Values.kafka.topic }}
        lagThreshold: "500"
        version: 2.8.1
        sasl: scram_sha512

微服务Deployment配置 #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ $.Chart.Name }}
  labels:
    app: {{ $.Chart.Name }}
    component: {{ $.Chart.Name }}
    release: {{ $.Release.Name }}
spec:
  replicas: {{ .Values.deployment.replicas }}
  strategy:
    type: RollingUpdate
    rollingUpdate:
      {{- if .Values.deployment.rollingUpdate.enabled }}
      maxSurge: {{ $.Values.deployment.rollingUpdate.maxSurge }}
      maxUnavailable: {{ $.Values.deployment.rollingUpdate.maxUnavailable }}
      {{- end }}
  selector:
    matchLabels:
      app: {{ $.Chart.Name }}
      release: {{ $.Release.Name }}
  template:
    metadata:
      labels:
        app: {{ $.Chart.Name }}
        version: {{ $.Chart.Version }}
        release: {{ $.Release.Name }}
    spec:
      imagePullSecrets:
      - name: {{ $.Values.deployment.image.imagepullsecret }}
      containers:
      - name: {{ $.Chart.Name }}
        image: "{{ .Values.deployment.image.repository }}:{{ .Values.deployment.image.tag }}"
        ports:
        - name: {{ $.Chart.Name }}
          containerPort: {{ $.Values.deployment.ports.http }}
          protocol: {{ $.Values.service.protocol }}
        imagePullPolicy: {{ $.Values.deployment.image.pullPolicy | quote }}
        resources:
          requests:
            memory: "{{ .Values.deployment.resources.requests.memory }}"
            cpu: "{{ .Values.deployment.resources.requests.cpu }}"
          limits:
            memory: "{{ .Values.deployment.resources.limits.memory }}"
            cpu: "{{ .Values.deployment.resources.limits.cpu }}"
        {{- if $.Values.deployment.volumes.enabled }}
        volumeMounts:
        - name: volume
          mountPath: /var/volume/myapp
        {{- end }}
        {{- if $.Values.deployment.env }}
        env:
          {{- range .Values.deployment.env }}
          - name: {{ .name }}
            value: "{{ .value }}"
          {{- end }}
        {{- end }}
      {{- if $.Values.deployment.volumes.enabled }}
      volumes:
      - name: volume
        emptyDir: {}
      {{- end }}
      {{- if $.Values.deployment.nodeSelector }}
      nodeSelector:
        {{- toYaml $.Values.deployment.nodeSelector | nindent 8 }}
      {{- end }}

错误日志 #

KEDA Operator Pod中出现以下错误：

ERROR   scale_handler   error resolving auth params
ERROR   Error getting scalers
ERROR   Failed to create new HPA resource
Warning  KEDAScalerFailed         13m (x82 over 18h)  keda-operator  error creating kafka client: kafka: client has run out of available brokers to talk to
Warning  ScaledObjectCheckFailed  13m (x82 over 18h)  keda-operator  failed to ensure HPA is correctly created for ScaledObject

可能的问题原因 #

• 网络连通性故障：KEDA Operator所在的keda命名空间Pod无法访问MSK集群的broker端口。需检查：
  • MSK集群的安全组是否允许EKS集群节点的IP段访问对应端口（SASL/SCRAM默认用9096端口）
  • MSK与EKS是否处于同一VPC，或已配置VPC对等连接/中转网关
  • BootstrapServers是否使用MSK的内网地址（若为私网集群），而非公网地址（若未开启公网访问）
• 认证配置不完整：ScaledObject的Kafka触发器缺少必要的安全协议配置，默认使用PLAINTEXT无法匹配MSK的SASL/SCRAM认证要求。需添加securityProtocol: SASL_SSL到触发器metadata中
• RBAC权限不足：KEDA Operator的ServiceAccount没有读取default命名空间下Secret和TriggerAuthentication资源的权限。需为该ServiceAccount添加对应命名空间的get、list权限
• Secret数据错误：Secret中的username和password存在Base64编码错误（如重复编码），或与MSK中配置的SASL/SCRAM credentials不匹配
• Kafka版本不兼容：ScaledObject中指定的version: 2.8.1与MSK集群的实际Kafka版本不匹配，导致客户端连接失败
• BootstrapServers配置错误：配置的broker地址或端口不正确，需确认是否为MSK控制台提供的正确SASL/SCRAM连接地址

内容的提问来源于stack exchange，提问作者Maor agai