HDFS节点切换后，如何调整固定域名端口的文件上传代码？

针对你当前代码中HDFS地址硬编码、节点切换后无法适配的问题，以下是几个实用的解决方案：

1. 用配置文件管理HDFS连接信息 #

把HDFS的URL、用户名等配置项抽离到单独的配置文件中，节点切换时只需修改配置文件，无需改动代码。

示例实现： #

首先创建一个config.ini配置文件：

[HDFS]
url = http://hdfs_server:50070
user = dr.who

然后修改代码读取配置：

from hdfs import InsecureClient
import configparser
import logging

def upload_file_to_hdfs(local_path, remote_path):
    # 读取配置文件
    config = configparser.ConfigParser()
    config.read('config.ini')
    hdfs_url = config.get('HDFS', 'url')
    hdfs_user = config.get('HDFS', 'user')
    
    client = InsecureClient(url=hdfs_url, user=hdfs_user)
    try:
        ret = client.upload(remote_path, local_path)
        print(f'File is uploaded: {local_path} -> {remote_path}')
    except Exception as e:
        logging.error(f'Upload failed: {e}')
        ret = None
    return ret

2. 实现多节点自动重试/故障转移 #

维护一个HDFS候选节点列表，当当前节点连接失败时，自动尝试列表中的下一个节点，适配节点切换场景。

示例实现： #

from hdfs import InsecureClient
import logging

def upload_file_to_hdfs(local_path, remote_path):
    # 候选HDFS节点列表，可从配置文件读取
    hdfs_candidates = [
        'http://hdfs_node1:50070',
        'http://hdfs_node2:50070',
        'http://hdfs_node3:50070'
    ]
    hdfs_user = 'dr.who'
    ret = None
    
    for hdfs_url in hdfs_candidates:
        try:
            client = InsecureClient(url=hdfs_url, user=hdfs_user)
            ret = client.upload(remote_path, local_path)
            print(f'File is uploaded via {hdfs_url}: {local_path} -> {remote_path}')
            break  # 连接成功则退出循环
        except Exception as e:
            logging.warning(f'Failed to connect to {hdfs_url}: {e}')
    
    if ret is None:
        logging.error('All HDFS nodes are unavailable, upload failed')
    return ret

3. 通过环境变量传递连接信息 #

将HDFS地址和用户名设置为系统环境变量，代码中读取环境变量的值，这样在部署或节点切换时，只需修改环境变量即可。

示例实现： #

from hdfs import InsecureClient
import logging
import os

def upload_file_to_hdfs(local_path, remote_path):
    # 从环境变量读取配置，设置默认值做容错处理
    hdfs_url = os.getenv('HDFS_URL', 'http://default_hdfs:50070')
    hdfs_user = os.getenv('HDFS_USER', 'dr.who')
    
    client = InsecureClient(url=hdfs_url, user=hdfs_user)
    try:
        ret = client.upload(remote_path, local_path)
        print(f'File is uploaded: {local_path} -> {remote_path}')
    except Exception as e:
        logging.error(f'Upload failed: {e}')
        ret = None
    return ret

额外提示： #

• 原代码中缺少import logging，记得补充，否则会触发报错
• 生产环境建议优先采用配置文件+故障转移的组合方案，兼顾灵活性和可靠性

内容的提问来源于stack exchange，提问作者Павел Иванов