EKS环境下使用cqlsh连接Cassandra Pod时遭遇认证失败问题

问题概述 #

在EKS的observability2命名空间部署Jaeger和Cassandra后，所有Pod、Service状态正常，但无法通过cqlsh连接Cassandra：

• 无认证连接返回AuthenticationFailed('Remote end requires authentication')
• 使用默认账号密码cassandra/cassandra连接提示凭据错误
• 已在cassandra-values.yaml中配置dbUser.cassandraConfig.authenticator: AllowAllAuthenticator以跳过认证，但问题未解决

环境资源状态 #

raj@log-Pro observability_new % kubectl get all
NAME                                    READY   STATUS      RESTARTS        AGE
pod/cassandra-0                         1/1     Running     0               55m
pod/cassandra-1                         1/1     Running     0               53m
pod/cassandra-2                         1/1     Running     0               51m
pod/jaeger-cassandra-0                  1/1     Running     0               4h16m
pod/jaeger-cassandra-1                  1/1     Running     0               4h14m
pod/jaeger-cassandra-2                  1/1     Running     0               4h12m
pod/jaeger-cassandra-schema-mbgvs       0/1     Completed   1               4h16m
pod/jaeger-collector-6fcff8cc55-8h29m   1/1     Running     5 (4h14m ago)   4h16m
pod/jaeger-collector-6fcff8cc55-hwxzr   1/1     Running     5 (4h14m ago)   4h16m
pod/jaeger-query-7d75fbb5cd-5plkp       2/2     Running     5 (4h15m ago)   4h16m

NAME                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                        AGE
service/cassandra            ClusterIP   None             <none>        9042/TCP                                       55m
service/cassandra-headless   ClusterIP   None             <none>        7000/TCP,7001/TCP,7199/TCP,9042/TCP            55m
service/jaeger-cassandra     ClusterIP   None             <none>        7000/TCP,7001/TCP,7199/TCP,9042/TCP,9160/TCP   4h16m
service/jaeger-collector     ClusterIP   10.100.167.231   <none>        14250/TCP,14268/TCP,14269/TCP                  4h16m
service/jaeger-query         ClusterIP   10.100.178.73    <none>        80/TCP,16685/TCP,16687/TCP                     4h16m

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/jaeger-collector   2/2     2            2           4h16m
deployment.apps/jaeger-query       1/1     1            1           4h16m

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/jaeger-collector-6fcff8cc55   2         2         2       4h16m
replicaset.apps/jaeger-query-7d75fbb5cd       1         1         1       4h16m

NAME                                READY   AGE
statefulset.apps/cassandra          3/3     55m
statefulset.apps/jaeger-cassandra   3/3     4h16m

NAME                                COMPLETIONS   DURATION   AGE
job.batch/jaeger-cassandra-schema   1/1           2m8s       4h16m

cqlsh连接错误输出 #

raj@logPro observability_new % kubectl exec -it pod/cassandra-0 -n observability2 -- cqlsh

Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042': AuthenticationFailed('Remote end requires authentication')})
command terminated with exit code 1
raj@logPro observability_new % kubectl exec -it pod/cassandra-0 -n observability2 -- cqlsh cassandra.observability2.svc.cluster.local 9042 -u cassandra -p cassandra


Warning: Using a password on the command line interface can be insecure.
Recommendation: use the credentials file to securely provide the password.

Connection error: ('Unable to connect to any servers', {'172.16.22.42:9042': AuthenticationFailed('Failed to authenticate to 172.16.22.42:9042: Error from server: code=0100 [Bad credentials] message="Provided username cassandra and/or password are incorrect"'), '172.16.33.113:9042': AuthenticationFailed('Failed to authenticate to 172.16.33.113:9042: Error from server: code=0100 [Bad credentials] message="Provided username cassandra and/or password are incorrect"'), '172.16.22.201:9042': AuthenticationFailed('Failed to authenticate to 172.16.22.201:9042: Error from server: code=0100 [Bad credentials] message="Provided username cassandra and/or password are incorrect"')})
command terminated with exit code 1

cassandra-values.yaml配置 #

global:
  imageRegistry: ""
  imagePullSecrets: []
  defaultStorageClass: ""
nameOverride: ""
fullnameOverride: ""
clusterDomain: cluster.local
extraDeploy: []
diagnosticMode:
  enabled: false
  command:
    - sleep
  args:
    - infinity
image:
  registry: docker.io
  repository: bitnami/cassandra
  tag: 5.0.2-debian-12-r3
  pullPolicy: IfNotPresent
  pullSecrets: []
  debug: false
dbUser:
  user: cassandra
  password: cassandra
  cassandraConfig:
    authenticator: AllowAllAuthenticator
  existingSecret: ""
initdbScripts:
  create-jaeger-keyspace.cql: |
    CREATE KEYSPACE IF NOT EXISTS jaeger_v1_dc1
    WITH replication = {
      'class': 'NetworkTopologyStrategy',
      'datacenter1': 3
    };
    USE jaeger_v1_dc1;
    CREATE TABLE IF NOT EXISTS traces (
      trace_id text,
      timestamp bigint,
      span_id text,
      PRIMARY KEY ((trace_id), timestamp)
    ) WITH CLUSTERING ORDER BY (timestamp DESC);
cluster:
  name: cassandra
  seedCount: 1
  numTokens: 256
  datacenter: datacenter1
  rack: rack1
  endpointSnitch: GossipingPropertyFileSnitch
  clientEncryption: false
  extraSeeds: []
  enableUDF: false
jvm:
  extraOpts: "-Dcassandra.ignore_dc=true"
  maxHeapSize: ""
  newHeapSize: ""
extraEnvVars:
  - name: JAEGER_ENDPOINT
    value: http://jaeger-collector.observability2.svc.cluster.local:14250
  - name: CASSANDRA_DATACENTER
    value: datacenter1
  - name: CASSANDRA_RACK
    value: rack1
replicaCount: 3
updateStrategy:
  type: RollingUpdate
podManagementPolicy: OrderedReady
resourcesPreset: "large"
resources: {}
livenessProbe:
  enabled: true
  initialDelaySeconds: 60
  periodSeconds: 30
  timeoutSeconds: 30
  successThreshold: 1
  failureThreshold: 5
readinessProbe:
  enabled: true
  exec:
    command:
      - /bin/bash
      - -ec
      - nodetool status | grep -E "^UN\s+${POD_IP}"
  initialDelaySeconds: 90
  periodSeconds: 30
  timeoutSeconds: 30
  failureThreshold: 5
  successThreshold: 1
startupProbe:
  enabled: false
persistence:
  enabled: true
  accessModes:
    - ReadWriteOnce
  size: 8Gi
  mountPath: /bitnami/cassandra
  commitLogMountPath: ""
  annotations: {}
service:
  type: ClusterIP
  ports:
    cql: 9042
    metrics: 8080
  clusterIP: None
  sessionAffinity: None
  annotations: {}
networkPolicy:
  enabled: true
  allowExternal: true
  allowExternalEgress: true
pdb:
  create: true
  minAvailable: ""
  maxUnavailable: ""
metrics:
  enabled: false
tls:
  internodeEncryption: none
  clientEncryption: false
  autoGenerated: false
volumePermissions:
  enabled: false
affinity: {}
nodeSelector: {}
tolerations: []
topologySpreadConstraints: []
podSecurityContext:
  enabled: true
  fsGroup: 1001
  fsGroupChangePolicy: Always
containerSecurityContext:
  enabled: true
  runAsUser: 1001
  runAsGroup: 1001
  runAsNonRoot: true
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]
  seccompProfile:
    type: RuntimeDefault
  readOnlyRootFilesystem: true
serviceAccount:
  create: true
  automountServiceAccountToken: false
lifecycleHooks: {}
terminationGracePeriodSeconds: ""
schedulerName: ""
initContainers: []
sidecars: []
extraVolumes: []
extraVolumeMounts: []

排查与解决建议 #

• 验证配置是否实际生效
  进入Cassandra Pod查看真实的cassandra.yaml配置：

  kubectl exec -it cassandra-0 -n observability2 -- cat /opt/bitnami/cassandra/conf/cassandra.yaml | grep authenticator
  

  如果输出不是AllowAllAuthenticator，说明配置未正确应用：

  • 检查helm部署命令是否正确传入了该values文件
  • 由于persistence.enabled=true，持久化卷可能保留了旧配置，需删除对应PVC后重新部署

• 检查初始化日志
  查看Cassandra初始化容器的日志，确认是否有修改认证配置或用户的操作：

  kubectl logs cassandra-0 -n observability2 -c initdb
  

• 重置超级用户密码（若认证为PasswordAuthenticator）
  如果实际配置仍是PasswordAuthenticator，尝试用nodetool重置密码：

  kubectl exec -it cassandra-0 -n observability2 -- nodetool resetpassword cassandra
  

  重置后重新尝试连接

• 确认连接目标实例
  注意环境中存在两个Cassandra StatefulSet（cassandra和jaeger-cassandra），确保连接的是目标实例，可直接指定Pod本地IP连接：

  kubectl exec -it cassandra-0 -n observability2 -- cqlsh 127.0.0.1 9042
  

• 检查环境变量覆盖
  查看Pod环境变量是否存在覆盖配置的情况：

  kubectl exec -it cassandra-0 -n observability2 -- env | grep CASSANDRA_AUTHENTICATOR
  

  若该变量存在且值不为AllowAllAuthenticator，会覆盖配置文件设置，需移除或修改该环境变量

内容的提问来源于stack exchange，提问作者Raj