本地Docker部署HDFS后Spark访问HDFS连接超时问题解决

问题背景 #

通过Docker Compose部署Hadoop环境，计划用HDFS存储文件，本地编写Spark Scala程序访问HDFS时，出现连接超时错误。

Docker Compose配置 #

services:
namenode:
image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8
container_name: namenode
restart: always
ports:
- 9870:9870
- 9000:9000
volumes:
- hadoop_namenode:/hadoop/dfs/name
- ./hadoop_namenode:/hadoop/dfs/data
environment:
- CLUSTER_NAME=test
env_file:
- ./hadoop.env

datanode:
image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8
container_name: datanode
restart: always
volumes:
- hadoop_datanode:/hadoop/dfs/data
environment:
SERVICE_PRECONDITION: "namenode:9870"
env_file:
- ./hadoop.env

resourcemanager:
image: bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8
container_name: resourcemanager
restart: always
environment:
SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864"
env_file:
- ./hadoop.env

nodemanager1:
image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8
container_name: nodemanager
restart: always
environment:
SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864 resourcemanager:8088"
env_file:
- ./hadoop.env

historyserver:
image: bde2020/hadoop-historyserver:2.0.0-hadoop3.2.1-java8
container_name: historyserver
restart: always
environment:
SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864 resourcemanager:8088"
volumes:
- hadoop_historyserver:/hadoop/yarn/timeline
env_file:
- ./hadoop.env
volumes:
hadoop_namenode:
hadoop_datanode:
hadoop_historyserver:

Spark Scala代码 #

package com.example.spark

import org.apache.spark.sql._

object Test {
def main(args: Array[String]): Unit = {
val spark = SparkSession
.builder
.appName("Test")
.config("fs.defaultFS", "hdfs://localhost:9000")
.master("local[*]")
.getOrCreate()

//    val df = spark.read.json("file:///C:/Users/user/Documents/dev/docker/hadoop_namenode") // NOTE: it reads successfully from the filesystem, just not HDFS.
val df = spark.read.json("hdfs://localhost:9000/app/sports/nba/games")
df.show(false)
}
}

错误信息 #

java.net.ConnectException: Connection timed out: no further information
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)...

[Executor task launch worker for task 0] WARN org.apache.hadoop.hdfs.DFSClient - Failed to connect to /172.20.0.3:9866 for file /app/sports/nba/games/sample_game.json for block BP-2085377089-172.20.0.7-1676688130925:blk_1073741882_1058, add to deadNodes and continue.
java.net.ConnectException: Connection timed out: no further information
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)...

[Executor task launch worker for task 0] WARN org.apache.hadoop.hdfs.DFSClient - No live nodes contain block BP-2085377089-172.20.0.7-1676688130925:blk_1073741882_1058 after checking nodes = [DatanodeInfoWithStorage[172.20.0.3:9866,DS-81d2fe5a-74e5-43cc-a2c6-1d56e5f89821,DISK]], ignoredNodes = null
[Executor task launch worker for task 0] INFO org.apache.hadoop.hdfs.DFSClient - Could not obtain BP-2085377089-172.20.0.7-1676688130925:blk_1073741882_1058 from any node:  No live nodes contain current block Block locations: DatanodeInfoWithStorage[172.20.0.3:9866,DS-81d2fe5a-74e5-43cc-a2c6-1d56e5f89821,DISK] Dead nodes:  DatanodeInfoWithStorage[172.20.0.3:9866,DS-81d2fe5a-74e5-43cc-a2c6-1d56e5f89821,DISK]. Will get new block locations from namenode and retry...
[Executor task launch worker for task 0] WARN org.apache.hadoop.hdfs.DFSClient - DFS chooseDataNode: got # 1 IOException, will wait for 1444.1894602927216 msec.
[Executor task launch worker for task 0] WARN org.apache.hadoop.hdfs.client.impl.BlockReaderFactory - I/O error constructing remote block reader.
java.net.ConnectException: Connection timed out: no further information
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
...

问题原因 #

从错误日志可见，Spark程序能正常连接Namenode（9000端口），但尝试连接Datanode的容器内网IP（172.20.0.3）和9866数据传输端口时超时，核心原因有两点：

1. Datanode向Namenode注册的是容器内网IP，本地Spark程序无法直接访问Docker容器内网网络
2. Docker Compose配置中仅映射了Namenode的端口，未映射Datanode的9866数据传输端口和9864管理端口

解决步骤 #

步骤1：修改Docker Compose，添加Datanode端口映射 #

在datanode服务中新增ports字段，映射必要端口：

datanode:
image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8
container_name: datanode
restart: always
ports:
- 9864:9864  # Datanode HTTP管理端口
- 9866:9866  # Datanode数据传输端口
volumes:
- hadoop_datanode:/hadoop/dfs/data
environment:
SERVICE_PRECONDITION: "namenode:9870"
env_file:
- ./hadoop.env

步骤2：配置Hadoop服务的对外访问规则 #

编辑hadoop.env文件，添加以下环境变量，确保Datanode能被本地程序访问：

CORE_CONF_fs_defaultFS=hdfs://namenode:9000
CORE_CONF_hadoop_http_staticuser_user=root
CORE_CONF_hadoop_proxyuser_hue_hosts=*
CORE_CONF_hadoop_proxyuser_hue_groups=*
HDFS_CONF_dfs_webhdfs_enabled=true
HDFS_CONF_dfs_permissions_enabled=false
# 关键配置：允许Datanode绑定所有网卡，关闭IP/主机名检查
HDFS_CONF_dfs_datanode_address=0.0.0.0:9866
HDFS_CONF_dfs_datanode_http_address=0.0.0.0:9864
HDFS_CONF_dfs_namenode_datanode_registration_ip___hostname___check=false

步骤3：重启Hadoop集群 #

执行命令重启所有容器，让配置生效：

docker-compose down
docker-compose up -d

步骤4：验证配置与运行程序 #

1. 访问Namenode管理页面http://localhost:9870，进入Datanodes页面确认Datanode状态正常
2. 重新运行Spark程序，此时应能正常连接Datanode读取HDFS文件

额外注意事项 #

• 若在远程服务器部署Hadoop，需将localhost替换为服务器公网IP，并确保防火墙开放9000、9870、9864、9866端口
• 可通过docker exec -it namenode hdfs dfs -ls /app/sports/nba/games命令，验证HDFS目标路径是否存在文件

内容的提问来源于stack exchange，提问作者MDrapic