使用Python与Docker Compose从Hadoop Namenode下载文件时遇连接错误

环境配置 #

API服务的docker-compose.yml #

version: "2.12"

services:
    api:
        container_name: api
        build:
            context: .
            dockerfile: Dockerfile
        environment:
            - BROKER_URI=redis://redis
            - BACKEND_URI=redis://redis
        depends_on:
            - worker
            - redis
    worker:
        container_name: celery
        environment:
            - BROKER_URI=redis://redis
            - BACKEND_URI=redis://redis
        build:
            context: .
            dockerfile: Dockerfile_worker
        depends_on:
            - redis
    redis:
        image: "redis:alpine3.17"
        ports:
            - 6379

networks:
    default:
        external:
            name: rete1

Hadoop集群的docker-compose.yml #

version: "3"

services:
  namenode:
    image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8
    container_name: namenode
    restart: always
    ports:
      - 9870:9870
      - 9000:9000
    volumes:
      - hadoop_namenode:/hadoop/dfs/name
    environment:
      - CLUSTER_NAME=test
    env_file:
      - ./hadoop.env

  datanode:
    image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8
    container_name: datanode
    restart: always
    hostname: localhost
    ports:
      - 9864:9864
    volumes:
      - hadoop_datanode:/hadoop/dfs/data
    environment:
      SERVICE_PRECONDITION: "namenode:9870"
    env_file:
      - ./hadoop.env
  
  resourcemanager:
    image: bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8
    container_name: resourcemanager
    restart: always
    environment:
      SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864"
    env_file:
      - ./hadoop.env

  nodemanager1:
    image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8
    container_name: nodemanager
    restart: always
    environment:
      SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864 resourcemanager:8088"
    env_file:
      - ./hadoop.env
  
  historyserver:
    image: bde2020/hadoop-historyserver:2.0.0-hadoop3.2.1-java8
    container_name: historyserver
    restart: always
    environment:
      SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864 resourcemanager:8088"
    volumes:
      - hadoop_historyserver:/hadoop/yarn/timeline
    env_file:
      - ./hadoop.env
  
volumes:
  hadoop_namenode:
  hadoop_datanode:
  hadoop_historyserver:


networks:
    default:
        external:
            name: rete1

测试场景与代码实现 #

已手动将文件上传至HDFS的/user/dataset/prova.csv路径，所有者为root。通过docker network inspect rete1确认：

• Namenode容器IP：172.21.0.5/16，监听9870端口
• API容器IP：172.21.0.9:5000

文件下载逻辑（test.py） #

import hdfs

HDFS_HOST = "http://172.21.0.5:9870"
client = hdfs.InsecureClient(HDFS_HOST, timeout=60)

def foo(content_url):
    client.download(hdfs_path=content_url, local_path=".")

API接口代码（api.py） #

from flask import Flask
from flask_restx import Api, Resource
from test import foo
from flask_restx import reqparse

api = Api(title="API", version="0.1")
application = Flask(__name__)
api.init_app(application)

parser = reqparse.RequestParser()
parser.add_argument('content_url', type=str, required=True)

@api.route("/my_endpoint/preprocess", methods=['POST'])
class Preprocess(Resource):
    @api.expect(parser)
    def post(self):
        params = parser.parse_args()
        foo(params['content_url'])

测试代码 #

import requests
requests.post("http://172.21.0.9:5000/my_endpoint/preprocess",
              json={"content_url": '/user/dataset/prova.csv'})

错误信息 #

requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=9864): Max retries exceeded with url: /webhdfs/v1/user/root/unprocessed.csv?op=OPEN&user.name=root&namenoderpcaddress=namenode:9000&offset=0 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5f4c7f01c0>: Failed to establish a new connection: [Errno 111] Connection refused'))

问题分析与解决方案 #

问题根源 #

1. Datanode主机名配置错误：Hadoop集群的datanode容器设置了hostname: localhost，当namenode处理文件下载请求时，会返回datanode的主机名（localhost）给API客户端。API容器内的localhost指向自身，而非datanode容器，导致连接被拒绝。
2. HDFS客户端未指定用户：文件所有者是root，但客户端未明确指定用户，可能导致权限或地址解析异常。
3. 硬编码IP不稳定：直接使用namenode的IP地址，不如使用容器名（同一Docker网络内可直接解析）可靠。

解决步骤 #

1. 修改Hadoop集群的datanode配置

编辑Hadoop的docker-compose.yml，修改datanode部分：

• 移除hostname: localhost，或设置为容器名hostname: datanode
• 编辑hadoop.env文件，添加以下配置：

  DATANODE_HOST=datanode
  HDFS_DATANODE_ADDRESS=0.0.0.0:9864
  HDFS_DATANODE_HTTP_ADDRESS=0.0.0.0:9864
  

  这会让datanode绑定到所有网卡，且namenode返回的datanode地址为容器名datanode，API容器可通过Docker网络解析到正确IP。

2. 修改HDFS客户端配置（test.py）

用容器名代替硬编码IP，并指定用户为root：

import hdfs

# 使用容器名namenode，同一网络内可直接解析
HDFS_HOST = "http://namenode:9870"
# 指定用户为root，匹配文件所有者
client = hdfs.InsecureClient(HDFS_HOST, user='root', timeout=60)

def foo(content_url):
    client.download(hdfs_path=content_url, local_path=".")

3. 重启服务

• 重启Hadoop集群：

  docker-compose down
  docker-compose up -d
  

• 重启API服务：

  docker-compose restart api
  

验证 #

重新运行测试代码，即可正常从HDFS下载文件。

内容的提问来源于stack exchange，提问作者Bob