Strimzi Connect Operator未创建plugins文件夹问题求助

1. Plugins目录无法持久化：Kubernetes Pod为临时容器，手动创建目录、复制Jar后重启Pod会丢失所有修改，因为容器镜像本身不包含该目录，也未通过持久化存储挂载。
2. KafkaConnect配置结构错误：externalConfiguration字段未正确定义卷挂载规则，导致指定的plugin.path无法关联到存储资源。
3. ConfigMap不适合存储Jar文件：ConfigMap设计用于文本配置存储，二进制Jar文件存入后会出现读取异常，需改用更合适的存储方案。

1. 修正KafkaConnect资源配置 #

推荐两种稳定方案解决Jar持久化问题，按需选择：

方案一：自定义包含JDBC Connector的镜像（Strimzi官方推荐） #

通过构建自定义镜像，将JDBC Connector和PostgreSQL驱动内置，避免容器重启后Jar丢失：

• 编写Dockerfile：

FROM strimzi/kafka-connect:latest

# 创建插件目录并下载JDBC Connector、PostgreSQL驱动
RUN mkdir -p /opt/kafka/external-configuration/plugins/jdbc && \
    curl -L https://d1i4a15mxbxib1.cloudfront.net/api/plugins/confluentinc/kafka-connect-jdbc/versions/10.7.6/confluentinc-kafka-connect-jdbc-10.7.6.zip -o jdbc.zip && \
    unzip jdbc.zip -d /opt/kafka/external-configuration/plugins/jdbc && \
    rm jdbc.zip && \
    curl -L https://jdbc.postgresql.org/download/postgresql-42.6.0.jar -o /opt/kafka/external-configuration/plugins/jdbc/postgresql-42.6.0.jar

• 构建本地镜像：

docker build -t local-strimzi-connect-jdbc:latest .

• 修改KafkaConnect配置，使用自定义镜像：

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  namespace: strimzi
spec:
  replicas: 1
  bootstrapServers: my-kafka-cluster-kafka-bootstrap:9092
  config:
    group.id: "connect-cluster"
    offset.storage.topic: "connect-cluster-offsets"
    config.storage.topic: "connect-cluster-configs"
    status.storage.topic: "connect-cluster-status"
    plugin.path: "/opt/kafka/external-configuration/plugins"
  image: local-strimzi-connect-jdbc:latest
  template:
    pod:
      securityContext:
        runAsUser: 0

方案二：InitContainer+PVC持久化Jar文件 #

无需构建镜像，通过初始化容器下载Jar到持久化卷，挂载到Connect Pod：

• 先创建PVC：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: connect-plugins-pvc
  namespace: strimzi
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

• 修改KafkaConnect配置，添加InitContainer和PVC挂载：

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  namespace: strimzi
spec:
  replicas: 1
  bootstrapServers: my-kafka-cluster-kafka-bootstrap:9092
  config:
    group.id: "connect-cluster"
    offset.storage.topic: "connect-cluster-offsets"
    config.storage.topic: "connect-cluster-configs"
    status.storage.topic: "connect-cluster-status"
    plugin.path: "/opt/kafka/external-configuration/plugins"
  image: strimzi/kafka-connect:latest
  template:
    pod:
      securityContext:
        runAsUser: 0
      initContainers:
        - name: download-jdbc-plugin
          image: curlimages/curl:latest
          command:
            - sh
            - -c
            - |
              mkdir -p /opt/kafka/external-configuration/plugins/jdbc
              curl -L https://d1i4a15mxbxib1.cloudfront.net/api/plugins/confluentinc/kafka-connect-jdbc/versions/10.7.6/confluentinc-kafka-connect-jdbc-10.7.6.zip -o /tmp/jdbc.zip
              unzip /tmp/jdbc.zip -d /opt/kafka/external-configuration/plugins/jdbc
              curl -L https://jdbc.postgresql.org/download/postgresql-42.6.0.jar -o /opt/kafka/external-configuration/plugins/jdbc/postgresql-42.6.0.jar
          volumeMounts:
            - name: plugins-volume
              mountPath: /opt/kafka/external-configuration/plugins
  externalConfiguration:
    volumes:
      - name: plugins-volume
        persistentVolumeClaim:
          claimName: connect-plugins-pvc

2. 修正KafkaConnector配置 #

解决配置冲突问题，确保同步逻辑正常：

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: postgres-source-connector
  namespace: strimzi
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: io.confluent.connect.jdbc.JdbcSourceConnector
  tasksMax: 1
  config:
    connector.class: io.confluent.connect.jdbc.JdbcSourceConnector
    tasks.max: "1"
    topic.prefix: "postgres-"
    connection.url: "jdbc:postgresql://postgres:5432/exampledb"
    connection.user: "postgres"
    connection.password: "examplepassword"
    table.whitelist: "users"
    mode: "timestamp"  # 与timestamp.column.name匹配，移除冲突的bulk模式
    poll.interval.ms: "5000"
    timestamp.column.name: "last_modified"
    numeric.mapping: "best_fit"
    value.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter.schemas.enable: "false"

3. 应用配置并验证 #

• 应用所有修改：

kubectl apply -f pvc.yaml -n strimzi
kubectl apply -f kafka-connect.yaml -n strimzi
kubectl apply -f kafka-connector.yaml -n strimzi

• 检查Pod状态：

kubectl get pods -n strimzi

• 验证插件加载：

kubectl exec -it <connect-pod-name> -n strimzi -- curl http://localhost:8083/connector-plugins | grep JdbcSourceConnector

• 禁止手动修改容器内文件：Kubernetes Pod为临时资源，所有手动修改会随Pod重启丢失，必须通过镜像或持久化卷解决。
• 保持plugin.path与挂载路径一致：确保配置中的插件路径和实际挂载路径完全匹配，否则Connect无法加载插件。
• 单独下载PostgreSQL驱动：Confluent JDBC Connector不包含数据库驱动，需额外添加对应驱动Jar。

内容的提问来源于stack exchange，提问作者user1834664