Strimzi与AKS azurefile存储类兼容吗？删除主题遇权限错误

问题现象 #

在AKS环境部署Strimzi Kafka集群时，仅使用基于file.csi.azure.com驱动的azurefile存储类时，Kafka代理在需要删除/清理日志文件的场景下会触发异常，导致Pod崩溃且无法自行恢复，必须删除并重建集群。使用kubernetes.io/azure-disk或disk.csi.azure.com存储类则无此问题。

触发异常的场景：

• 删除Kafka主题时
• 采用compact清理策略的主题在日志清理器运行时

复现步骤 #

• 部署包含1个Kafka代理、1个ZooKeeper副本的Strimzi集群，存储类指定为azurefile
• 确认集群Pod正常运行，自定义主题处于就绪状态
• 执行kubectl delete kafkatopic custom-topic删除主题，或等待compact策略主题触发日志清理
• 观察到Kafka代理Pod崩溃

Strimzi集群配置 #

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: test-cluster
  namespace: kafka-test
spec:
  kafka:
    version: 3.0.0
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        type: nodeport
        tls: false
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.0"
      auto.create.topics.enable: "false"
    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          size: 2Gi
          deleteClaim: true
          class: azurefile
  zookeeper:
    replicas: 1
    storage:
      type: persistent-claim
      size: 2Gi
      deleteClaim: true
      class: azurefile
  entityOperator:
    topicOperator: {}
    userOperator: {}

Strimzi主题配置 #

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: custom-topic
  labels:
    strimzi.io/cluster: test-cluster
spec:
  partitions: 1
  replicas: 1

错误日志 #

删除主题后，代理Pod抛出的核心错误如下：

2022-07-19 10:00:16,211 ERROR Error while renaming dir for custom-topic-0 in log dir /var/lib/kafka/data-0/kafka-log0 (kafka.server.LogDirFailureChannel) [control-plane-kafka-request-handler-0]
java.nio.file.AccessDeniedException: /var/lib/kafka/data-0/kafka-log0/custom-topic-0 -> /var/lib/kafka/data-0/kafka-log0/custom-topic-0.20aa2754010549d58935ea4144c2f1f6-delete
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:478)
        at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:267)
        at java.base/java.nio.file.Files.move(Files.java:1422)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:932)
        at kafka.log.Log.$anonfun$renameDir$2(Log.scala:699)
        at kafka.log.Log.renameDir(Log.scala:2487)
        at kafka.log.LogManager.asyncDelete(LogManager.scala:1036)
        at kafka.log.LogManager.$anonfun$asyncDelete$3(LogManager.scala:1071)
        at scala.Option.foreach(Option.scala:437)
        at kafka.log.LogManager.$anonfun$asyncDelete$2(LogManager.scala:1069)
        at kafka.log.LogManager.$anonfun$asyncDelete$2$adapted(LogManager.scala:1067)
        at scala.collection.mutable.HashSet$Node.foreach(HashSet.scala:435)
        at scala.collection.mutable.HashSet.foreach(HashSet.scala:361)
        at kafka.log.LogManager.asyncDelete(LogManager.scala:1067)
        at kafka.server.ReplicaManager.stopPartitions(ReplicaManager.scala:468)
        at kafka.server.ReplicaManager.stopReplicas(ReplicaManager.scala:405)
        at kafka.server.KafkaApis.handleStopReplicaRequest(KafkaApis.scala:291)
        at kafka.server.KafkaApis.handle(KafkaApis.scala:174)
        at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:75)
        at java.base/java.lang.Thread.run(Thread.java:829)

尝试过的无效方案 #

创建自定义存储类，设置权限相关挂载参数，仍无法解决问题：

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: kafka-azurefile
provisioner: file.csi.azure.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - mfsymlinks
  - actimeo=30
  - uid=0
  - gid=0
parameters:
  skuName: Standard_LRS

环境版本 #

• Strimzi：0.27.1
• AKS Kubernetes：1.22.6

内容的提问来源于stack exchange，提问作者Andrew Savory