Kafka KRAFT模式下SCRAM-SHA-512认证失败求助

问题 #

在Kafka KRaft模式下配置SCRAM-SHA-512认证时无法正常工作，切换为PLAIN机制则功能正常。相关配置及错误信息如下：

Docker Compose配置 #

services:
  kafka-1:
    image: confluentinc/cp-kafka:7.7.1
    ports:
      - '9092:9092'
    environment:
      CLUSTER_ID: apCvyf6zusVSAvyqGAh7AZ
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:SASL_PLAINTEXT,CLIENT:SASL_PLAINTEXT,BROKER:SASL_PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'CLIENT://kafka-1:29092,BROKER://localhost:9092'
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:9093'
      KAFKA_LISTENERS: 'CLIENT://0.0.0.0:29092,CONTROLLER://0.0.0.0:9093,BROKER://0.0.0.0:9092'
      KAFKA_INTER_BROKER_LISTENER_NAME: BROKER
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL: SCRAM-SHA-512
      KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: SCRAM-SHA-512
      KAFKA_SASL_ENABLED_MECHANISMS: SCRAM-SHA-512
      KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer
      KAFKA_SUPER_USERS: 'User:admin'
      KAFKA_OPTS: '-Djava.security.auth.login.config=/etc/kafka/kafka_jaas.conf'
    volumes:
      - ./config/kafka_jaas.conf:/etc/kafka/kafka_jaas.conf 
      
  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    ports:
      - 1111:1111
    environment:
      SERVER_PORT: 1111
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka-1:29092
      KAFKA_CLUSTERS_0_ZOOKEEPER: zookeeper:2181
      KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_PLAINTEXT
      KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: SCRAM-SHA-512
      KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: 'org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="admin_password";'                

JAAS配置文件 #

KafkaServer {
  org.apache.kafka.common.security.scram.ScramLoginModule required
  username="admin"
  password="admin_password"
  user_admin="admin_password"
  user_service="service_password";
};

错误日志 #

kafka-1_1   | [2024-10-28 13:42:04,650] ERROR [controller-1-to-controller-registration-channel-manager]: Request ControllerRegistrationRequestData(controllerId=1, incarnationId=NZkn4xOFS42Kg0RhscJHVw, zkMigrationReady=false, listeners=[Listener(name='CONTROLLER', host='0.0.0.0', port=9093, securityProtocol=2)], features=[Feature(name='metadata.version', minSupportedVersion=1, maxSupportedVersion=19)]) failed due to authentication error with controller (kafka.server.NodeToControllerRequestThread)
kafka-1_1   | org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512
kafka-1_1   | [2024-10-28 13:42:04,650] ERROR [ControllerRegistrationManager id=1 incarnation=NZkn4xOFS42Kg0RhscJHVw] RegistrationResponseHandler: authentication error (kafka.server.ControllerRegistrationManager)
kafka-1_1   | org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512

解决方案 #

1. 修正JAAS配置文件 #

KRaft模式下，ScramLoginModule不需要在KafkaServer块中配置user_*字段（这是PLAIN机制的内存用户配置逻辑），只需保留broker/controller自身用于认证的账号密码即可。修改后的JAAS文件：

KafkaServer {
  org.apache.kafka.common.security.scram.ScramLoginModule required
  username="admin"
  password="admin_password";
};

2. 初始化SCRAM用户到Kafka元数据 #

SCRAM用户必须存储在Kafka的元数据中，不能仅依赖JAAS配置。启动Kafka集群后，执行以下命令创建用户：

# 进入kafka容器
docker exec -it kafka-1 bash

# 创建admin用户，指定SCRAM-SHA-512机制
kafka-configs.sh --bootstrap-server localhost:9092 \
  --command-config /etc/kafka/kafka_jaas.conf \
  --alter --add-config 'SCRAM-SHA-512=[password=admin_password]' \
  --entity-type users --entity-name admin

# 按需创建service用户
kafka-configs.sh --bootstrap-server localhost:9092 \
  --command-config /etc/kafka/kafka_jaas.conf \
  --alter --add-config 'SCRAM-SHA-512=[password=service_password]' \
  --entity-type users --entity-name service

执行命令前确保Kafka集群已正常启动，且--command-config指向的JAAS文件包含admin的认证信息。

3. 调整Kafka UI的JAAS配置 #

Docker Compose环境变量中无需转义双引号，直接用单引号包裹配置内容即可：

KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: 'org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="admin_password";'

4. 验证Controller监听配置 #

当前配置中KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL: SCRAM-SHA-512是正确的，只需确保controller节点使用的admin账号已成功写入元数据即可。

内容的提问来源于stack exchange，提问作者Casper T