基于Bitnami Flink Helm Chart添加外部依赖至JobManager与TaskManager

问题说明 #

使用Bitnami Flink Helm Chart编写values.yaml时，需要将mssql cdc、iceberg、nessie的JAR包加入Flink容器，确保Flink SQL Client能正确加载这些依赖，但不清楚具体操作方法。

之前的尝试 #

• 用初始化容器将JAR包下载到共享目录，但无法复制到Flink的/opt/bitnami/flink/lib目录
• 尝试覆盖JobManager启动命令，容器启动报错找不到config.yaml，相关配置如下：

jobmanager:
  initContainers:
    - name: download-flink-dependencies
      image: busybox:latest
      imagePullPolicy: IfNotPresent
      command:
        - /bin/sh
        - -c
        - |
          echo "Downloading dependencies..."
          DEPENDENCY_URL="https://repo.maven.apache.org/maven2/org/apache/iceberg/iceberg-flink-runtime-1.17/1.5.2/iceberg-flink-runtime-1.17-1.5.2.jar"
          wget -P /opt/flink/job $DEPENDENCY_URL
          echo "Dependencies downloaded."
      volumeMounts:
        - name: flink-job
          mountPath: /opt/flink/job
  extraVolumeMounts:
    - name: flink-job
      mountPath: /opt/flink/job
  extraVolumes:
    - name: flink-job
      emptyDir: {}
  extraEnvVars:
    - name: FLINK_CFG_STATE.CHECKPOINTS.DIR
      value: "s3p://flink/checkpoints"
    - name: FLINK_CFG_STATE_SAVEPOINTS_DIR
      value: "s3p://flink/savepoints"
    - name: FLINK_CFG_STATE_BACKEND_TYPE
      value: filesystem
    - name: FLINK_CFG_S3_AWS__ACCESS__KEY
      value: admin
    - name: FLINK_CFG_S3_AWS__SECRET__KEY
      value: password
    - name: FLINK_CFG_S3_ENDPOINT
      value: http://10.31.147.70:9000
    - name: FLINK_CFG_S3_PATH_STYLE_ACCESS
      value: "true"
    - name: HADOOP_CLASSPATH
      value: /opt/flink/job
  # command:
  #   - /bin/sh
  #   - -c
  #   - |
  #     chmod
  #     # Copy files to the Flink library path
  #     cp /opt/flink/job/*.jar /opt/bitnami/flink/lib/
  #     # Start the Flink JobManager
  #     /opt/bitnami/flink/bin/jobmanager.sh start
  resources:
    limits:
      cpu: 2
      memory: 2048Mi
    requests:
      cpu: 250m
      memory: 1024Mi

正确配置方案 #

核心思路 #

通过初始化容器下载依赖JAR到共享临时卷，然后修改JobManager、TaskManager、SQL Client的启动命令，先将JAR复制到Flink的lib目录，再执行Bitnami标准启动脚本（避免配置文件找不到的问题）。

完整values.yaml配置 #

# 全局配置，启用SQL Client（如果需要）
global:
  flink:
    sqlClient:
      enabled: true

jobmanager:
  initContainers:
    - name: download-deps
      image: busybox:latest
      imagePullPolicy: IfNotPresent
      command:
        - /bin/sh
        - -c
        - |
          # 下载所需依赖JAR，替换为对应版本的URL
          wget -P /tmp/deps https://repo.maven.apache.org/maven2/org/apache/iceberg/iceberg-flink-runtime-1.17/1.5.2/iceberg-flink-runtime-1.17-1.5.2.jar
          wget -P /tmp/deps https://repo.maven.apache.org/maven2/com/microsoft/sqlserver/mssql-jdbc/12.6.3.jre17/mssql-jdbc-12.6.3.jre17.jar
          wget -P /tmp/deps https://repo.maven.apache.org/maven2/org/projectnessie/nessie-flink-extensions/0.70.1/nessie-flink-extensions-0.70.1.jar
      volumeMounts:
        - name: flink-deps
          mountPath: /tmp/deps
  extraVolumeMounts:
    - name: flink-deps
      mountPath: /tmp/deps
  extraVolumes:
    - name: flink-deps
      emptyDir: {}
  # 修改启动命令：先复制JAR，再执行标准启动脚本
  command:
    - /bin/sh
    - -c
    - |
      cp /tmp/deps/*.jar /opt/bitnami/flink/lib/
      /opt/bitnami/scripts/flink/jobmanager/run.sh
  extraEnvVars:
    - name: FLINK_CFG_STATE.CHECKPOINTS.DIR
      value: "s3p://flink/checkpoints"
    - name: FLINK_CFG_STATE_SAVEPOINTS_DIR
      value: "s3p://flink/savepoints"
    - name: FLINK_CFG_STATE_BACKEND_TYPE
      value: filesystem
    - name: FLINK_CFG_S3_AWS__ACCESS__KEY
      value: admin
    - name: FLINK_CFG_S3_AWS__SECRET__KEY
      value: password
    - name: FLINK_CFG_S3_ENDPOINT
      value: http://10.31.147.70:9000
    - name: FLINK_CFG_S3_PATH_STYLE_ACCESS
      value: "true"
  resources:
    limits:
      cpu: 2
      memory: 2048Mi
    requests:
      cpu: 250m
      memory: 1024Mi

taskmanager:
  # TaskManager需同步配置，否则会出现依赖缺失
  initContainers:
    - name: download-deps
      image: busybox:latest
      imagePullPolicy: IfNotPresent
      command:
        - /bin/sh
        - -c
        - |
          wget -P /tmp/deps https://repo.maven.apache.org/maven2/org/apache/iceberg/iceberg-flink-runtime-1.17/1.5.2/iceberg-flink-runtime-1.17-1.5.2.jar
          wget -P /tmp/deps https://repo.maven.apache.org/maven2/com/microsoft/sqlserver/mssql-jdbc/12.6.3.jre17/mssql-jdbc-12.6.3.jre17.jar
          wget -P /tmp/deps https://repo.maven.apache.org/maven2/org/projectnessie/nessie-flink-extensions/0.70.1/nessie-flink-extensions-0.70.1.jar
      volumeMounts:
        - name: flink-deps
          mountPath: /tmp/deps
  extraVolumeMounts:
    - name: flink-deps
      mountPath: /tmp/deps
  extraVolumes:
    - name: flink-deps
      emptyDir: {}
  command:
    - /bin/sh
    - -c
    - |
      cp /tmp/deps/*.jar /opt/bitnami/flink/lib/
      /opt/bitnami/scripts/flink/taskmanager/run.sh
  resources:
    limits:
      cpu: 2
      memory: 2048Mi
    requests:
      cpu: 250m
      memory: 1024Mi

# SQL Client配置（如果需要使用SQL Client）
sqlClient:
  enabled: true
  initContainers:
    - name: download-deps
      image: busybox:latest
      imagePullPolicy: IfNotPresent
      command:
        - /bin/sh
        - -c
        - |
          wget -P /tmp/deps https://repo.maven.apache.org/maven2/org/apache/iceberg/iceberg-flink-runtime-1.17/1.5.2/iceberg-flink-runtime-1.17-1.5.2.jar
          wget -P /tmp/deps https://repo.maven.apache.org/maven2/com/microsoft/sqlserver/mssql-jdbc/12.6.3.jre17/mssql-jdbc-12.6.3.jre17.jar
          wget -P /tmp/deps https://repo.maven.apache.org/maven2/org/projectnessie/nessie-flink-extensions/0.70.1/nessie-flink-extensions-0.70.1.jar
      volumeMounts:
        - name: flink-deps
          mountPath: /tmp/deps
  extraVolumeMounts:
    - name: flink-deps
      mountPath: /tmp/deps
  extraVolumes:
    - name: flink-deps
      emptyDir: {}
  command:
    - /bin/sh
    - -c
    - |
      cp /tmp/deps/*.jar /opt/bitnami/flink/lib/
      /opt/bitnami/scripts/flink/sql-client/run.sh

注意事项 #

• 不要直接将emptyDir挂载到/opt/bitnami/flink/lib，会覆盖原有Flink核心依赖，导致服务启动失败
• 使用Bitnami提供的标准启动脚本（/opt/bitnami/scripts/flink/xxx/run.sh），确保配置文件能正确加载，避免之前的config.yaml找不到的错误
• JobManager、TaskManager、SQL Client必须同步配置依赖，否则部分组件会出现类找不到的问题

内容的提问来源于stack exchange，提问作者user27156312