基于Bitnami/Kafka镜像集成Aiven远程存储插件的问题求助

自定义Kafka镜像Dockerfile（集成Minio S3远程存储，基于Bitnami镜像+Aiven插件） #

FROM bitnami/kafka:3.9.0

USER root

RUN mkdir -p /opt/bitnami/kafka/plugins \
    &&  mkdir -p /opt/bitnami/kafka/tiered-storage-cache

RUN curl -L https://github.com/Aiven-Open/tiered-storage-for-apache-kafka/releases/download/2024-10-23-1729694047/core-0.0.1-SNAPSHOT.tgz -o /opt/bitnami/kafka/plugins/core.tgz \
    && curl -L https://github.com/Aiven-Open/tiered-storage-for-apache-kafka/releases/download/2024-10-23-1729694047/s3-0.0.1-SNAPSHOT.tgz -o /opt/bitnami/kafka/plugins/s3.tgz

RUN tar -xzf /opt/bitnami/kafka/plugins/core.tgz -C /opt/bitnami/kafka/plugins/ \
    && rm /opt/bitnami/kafka/plugins/core.tgz \
    && tar -xzf /opt/bitnami/kafka/plugins/s3.tgz -C /opt/bitnami/kafka/plugins/ \
    && rm /opt/bitnami/kafka/plugins/s3.tgz

RUN chown -R root:root /opt/bitnami/kafka \
    && chmod -R 777 /opt/bitnami/kafka/plugins /opt/bitnami/kafka/tiered-storage-cache
# Ensure that the subfolders in tiered-storage-cache will inherit the permissions
RUN chmod g+s /opt/bitnami/kafka/tiered-storage-cache

Kafka Kraft集群Broker配置 #

集群为3控制器+3Broker，Broker环境变量配置如下：

KAFKA_CFG_PROCESS_ROLES: "broker"
KAFKA_CFG_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"
KAFKA_CFG_SUPER_USERS: "User:sa"
KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_CFG_NUM_PARTITIONS: 12
KAFKA_CFG_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:SASL_PLAINTEXT,PLAINTEXT_HOST:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXT,CONTROLLER:SASL_PLAINTEXT"
KAFKA_CFG_SASL_ENABLED_MECHANISMS: "PLAIN"
KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL: "PLAIN"
KAFKA_CFG_SASL_MECHANISM_CONTROLLER_PROTOCOL: "PLAIN"
KAFKA_INTER_BROKER_USER: "sa"
KAFKA_INTER_BROKER_PASSWORD: "000000"
KAFKA_CONTROLLER_USER: "sa"
KAFKA_CONTROLLER_PASSWORD: "000000"
# CLIENT SETTINGS
KAFKA_CLIENT_LISTENER_NAME: "EXTERNAL"
KAFKA_CLIENT_USERS: "sa,da,va"
KAFKA_CLIENT_PASSWORDS: "000000,111111,222222"
KAFKA_CFG_LOG4J_LOGGERS: "io.aiven.kafka.tieredstorage=DEBUG,org.apache.kafka.common.security=DEBUG,org.apache.kafka.server.log.remote.metadata.storage=DEBUG" # Increase Tiered Storage log level
KAFKA_CFG_REMOTE_LOG_STORAGE_SYSTEM_ENABLE: "true" # Enable Tiered Storage
KAFKA_CFG_LOG_RETENTION_CHECK_INTERVAL_MS: 10000 # Log cleaner checks whether any log is eligible for deletion
KAFKA_CFG_REMOTE_LOG_MANAGER_TASK_INTERVAL_MS: 5000 # Interval at which remote log manager runs the scheduled tasks like copy segments, and clean up remote log segments.
# Remote metadata manager
KAFKA_CFG_REMOTE_LOG_METADATA_MANAGER_CLASS_NAME: "org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager"
KAFKA_CFG_REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME: "PLAINTEXT"
KAFKA_CFG_RLMM_CONFIG_REMOTE_LOG_METADATA_TOPIC_REPLICATION_FACTOR: 3
KAFKA_CFG_REMOTE_LOG_STORAGE_MANAGER_CLASS_PATH: "/opt/bitnami/kafka/plugins/core-0.0.1-SNAPSHOT/*:/opt/bitnami/kafka/plugins/s3-0.0.1-SNAPSHOT/*" 
KAFKA_CFG_REMOTE_LOG_STORAGE_MANAGER_CLASS_NAME: "io.aiven.kafka.tieredstorage.RemoteStorageManager"
KAFKA_CFG_RSM_CONFIG_CHUNK_SIZE: 4194304 # 4 MiB
KAFKA_CFG_RSM_CONFIG_FETCH_CHUNK_CACHE_CLASS: "io.aiven.kafka.tieredstorage.fetch.cache.DiskChunkCache"
KAFKA_CFG_RSM_CONFIG_FETCH_CHUNK_CACHE_PATH: "/opt/bitnami/kafka/tiered-storage-cache"
KAFKA_CFG_RSM_CONFIG_FETCH_CHUNK_CACHE_SIZE: 1073741824 # 1 GiB
KAFKA_CFG_RSM_CONFIG_FETCH_CHUNK_CACHE_PREFETCH_MAX_SIZE: 16777216 # 16 MiB
KAFKA_CFG_RSM_CONFIG_FETCH_CHUNK_CACHE_RETENTION_MS: 600000 # 600000 ms
KAFKA_CFG_RSM_CONFIG_CUSTOM_METADATA_FIELDS_INCLUDE: "REMOTE_SIZE"

遇到的问题 #

问题1：SASL配置导致自动创建主题失败 #

执行docker compose up时，本该自动创建的主题因SASL配置失败。

问题2：移除SASL后分层存储仍报错 #

移除SASL配置后自动创建主题正常，但创建带远程存储功能的主题时，报错：

ERROR org.apache.kafka.common.errors.InvalidConfigurationException: Tiered Storage functionality is disabled in the broker. Topic cannot be configured with remote log storage. (org.apache.kafka.tools.TopicCommand)

已确认Broker容器内server.properties中remote.log.storage.system.enable=true配置正确。

解决方案 #

针对问题1：SASL配置下自动创建主题失败 #

1. 临时调试可添加KAFKA_CFG_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"，跳过ACL检查验证自动创建逻辑是否正常；后续需为自动创建主题的请求用户配置对应ACL权限。
2. 检查Broker日志中SASL authentication failed相关条目，定位具体失败的用户，确认内部自动创建请求的凭证是否匹配KAFKA_INTER_BROKER_USER/KAFKA_INTER_BROKER_PASSWORD配置。
3. 确保监听器协议映射无冲突：当前所有监听器都使用SASL_PLAINTEXT，需确认Broker内部通信的监听器未被错误配置。

针对问题2：分层存储启用后仍报错 #

1. 修正插件目录权限：Bitnami Kafka默认以1001用户运行，需调整Dockerfile权限配置为：

   RUN chown -R 1001:1001 /opt/bitnami/kafka/plugins /opt/bitnami/kafka/tiered-storage-cache
   RUN chmod -R 755 /opt/bitnami/kafka/plugins
   RUN chmod g+s /opt/bitnami/kafka/tiered-storage-cache
   

   避免使用root权限运行插件目录，确保Broker进程能读取插件文件。
2. 补充Minio S3连接配置：Aiven插件缺少S3参数会初始化失败，导致Broker认为分层存储未启用，需添加：

   KAFKA_CFG_RSM_CONFIG_S3_ENDPOINT: "http://minio:9000"
   KAFKA_CFG_RSM_CONFIG_S3_ACCESS_KEY_ID: "minioadmin"
   KAFKA_CFG_RSM_CONFIG_S3_SECRET_ACCESS_KEY: "minioadmin"
   KAFKA_CFG_RSM_CONFIG_S3_BUCKET_NAME: "kafka-tiered-storage"
   KAFKA_CFG_RSM_CONFIG_S3_REGION: "us-east-1"
   KAFKA_CFG_RSM_CONFIG_S3_PATH_STYLE_ACCESS: "true"
   

3. 检查Broker启动日志：搜索Loaded remote storage manager或Failed to load remote storage manager关键字，确认插件是否成功加载；若有初始化错误日志，根据提示修复配置。

内容的提问来源于stack exchange，提问作者Quang Anh Trần