Docker环境下Flink与HDFS集成：HADOOP_CLASSPATH配置疑问

核心问题是官方Flink镜像默认不包含Hadoop类库，而Hadoop部署在独立容器中，需要让Flink容器能访问到Hadoop依赖并正确设置类路径，以下三种方法可解决：

方法一：构建包含Hadoop依赖的自定义Flink镜像 #

基于官方Flink镜像，安装和Hadoop容器同版本的Hadoop，直接在镜像中配置好HADOOP_CLASSPATH，容器启动时无需额外配置。

1. 创建Dockerfile：

FROM flink:1.17.2-scala_2.12-java8

# 安装wget用于下载Hadoop
RUN apt-get update && apt-get install -y wget --no-install-recommends

# 下载并解压Hadoop 3.2.1（和你的Hadoop容器版本一致）
RUN wget https://archive.apache.org/dist/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz && \
    tar -xzf hadoop-3.2.1.tar.gz -C /opt && \
    rm -f hadoop-3.2.1.tar.gz

# 设置和Hadoop容器一致的HADOOP_CLASSPATH
ENV HADOOP_CLASSPATH="/etc/hadoop:/opt/hadoop-3.2.1/share/hadoop/common/lib/*:/opt/hadoop-3.2.1/share/hadoop/common/*:/opt/hadoop-3.2.1/share/hadoop/hdfs:/opt/hadoop-3.2.1/share/hadoop/hdfs/lib/*:/opt/hadoop-3.2.1/share/hadoop/hdfs/*:/opt/hadoop-3.2.1/share/hadoop/mapreduce/lib/*:/opt/hadoop-3.2.1/share/hadoop/mapreduce/*:/opt/hadoop-3.2.1/share/hadoop/yarn:/opt/hadoop-3.2.1/share/hadoop/yarn/lib/*:/opt/hadoop-3.2.1/share/hadoop/yarn/*"

2. 构建自定义镜像：

docker build -t flink-with-hadoop:1.17.2 .

3. 修改docker-compose.yml中jobmanager和taskmanager的镜像为这个自定义镜像，去掉原来的HADOOP_CLASSPATH环境变量配置。

方法二：挂载Hadoop容器的类库目录到Flink容器 #

通过Docker卷挂载，把Hadoop容器里的Hadoop安装目录和配置目录挂载到Flink容器的对应路径，让Flink直接使用Hadoop的类库。

1. 查找Hadoop容器的实际目录路径：
   运行docker inspect namenode，查看Mounts字段下的源路径，比如/var/lib/docker/volumes/xxx/_data对应/opt/hadoop-3.2.1，/etc/hadoop的源路径同理。

2. 修改docker-compose.yml配置：

jobmanager:
  image: flink:1.17.2-scala_2.12-java8
  ports:
    - "8081:8081"
  volumes:
    # 挂载Hadoop容器的安装目录和配置目录到Flink容器
    - /path/to/namenode/opt/hadoop-3.2.1:/opt/hadoop-3.2.1
    - /path/to/namenode/etc/hadoop:/etc/hadoop
  environment:
    - HADOOP_CLASSPATH=/etc/hadoop:/opt/hadoop-3.2.1/share/hadoop/common/lib/*:/opt/hadoop-3.2.1/share/hadoop/common/*:/opt/hadoop-3.2.1/share/hadoop/hdfs:/opt/hadoop-3.2.1/share/hadoop/hdfs/lib/*:/opt/hadoop-3.2.1/share/hadoop/hdfs/*:/opt/hadoop-3.2.1/share/hadoop/mapreduce/lib/*:/opt/hadoop-3.2.1/share/hadoop/mapreduce/*:/opt/hadoop-3.2.1/share/hadoop/yarn:/opt/hadoop-3.2.1/share/hadoop/yarn/lib/*:/opt/hadoop-3.2.1/share/hadoop/yarn/*
    - |
      FLINK_PROPERTIES=
      ...

taskmanager:
  image: flink:1.17.2-scala_2.12-java8
  depends_on:
    - jobmanager
  scale: 2
  volumes:
    - /path/to/namenode/opt/hadoop-3.2.1:/opt/hadoop-3.2.1
    - /path/to/namenode/etc/hadoop:/etc/hadoop
  environment:
    - HADOOP_CLASSPATH=/etc/hadoop:/opt/hadoop-3.2.1/share/hadoop/common/lib/*:/opt/hadoop-3.2.1/share/hadoop/common/*:/opt/hadoop-3.2.1/share/hadoop/hdfs:/opt/hadoop-3.2.1/share/hadoop/hdfs/lib/*:/opt/hadoop-3.2.1/share/hadoop/hdfs/*:/opt/hadoop-3.2.1/share/hadoop/mapreduce/lib/*:/opt/hadoop-3.2.1/share/hadoop/mapreduce/*:/opt/hadoop-3.2.1/share/hadoop/yarn:/opt/hadoop-3.2.1/share/hadoop/yarn/lib/*:/opt/hadoop-3.2.1/share/hadoop/yarn/*
    - |
      FLINK_PROPERTIES=
      ...

替换/path/to/namenode/...为实际查到的路径即可。

方法三：使用Flink预编译的Hadoop兼容包 #

这是最简便的方法，Flink提供了包含所有Hadoop依赖的uber包，直接放到Flink容器的lib目录即可，无需设置HADOOP_CLASSPATH。

1. 下载对应版本的兼容包：
   选择匹配Flink 1.17.2、Hadoop 3.2.1的包，比如flink-shaded-hadoop-3-uber-3.2.1-10.0.jar。

2. 修改docker-compose.yml，将jar包挂载到Flink容器的lib目录：

jobmanager:
  image: flink:1.17.2-scala_2.12-java8
  ports:
    - "8081:8081"
  volumes:
    - ./flink-shaded-hadoop-3-uber-3.2.1-10.0.jar:/opt/flink/lib/flink-shaded-hadoop-3-uber-3.2.1-10.0.jar
  environment:
    - |
      FLINK_PROPERTIES=
      ...

taskmanager:
  image: flink:1.17.2-scala_2.12-java8
  depends_on:
    - jobmanager
  scale: 2
  volumes:
    - ./flink-shaded-hadoop-3-uber-3.2.1-10.0.jar:/opt/flink/lib/flink-shaded-hadoop-3-uber-3.2.1-10.0.jar
  environment:
    - |
      FLINK_PROPERTIES=
      ...

把jar包放在当前目录下，启动容器后Flink会自动加载这个包，无需额外配置类路径。

内容的提问来源于stack exchange，提问作者AnastaDol