从HDFS向非Hadoop集群远程服务器传输文件方案咨询

Alright, let's figure out how to get files from HDFS over to a remote server that's not part of your Hadoop cluster—without relying on curl or wget. I'll walk you through using rsyslog/syslog-ng first, then share some better tools that are actually built for file transfer (since syslog is really meant for logs, not moving files).

Keep in mind: Syslog tools are designed for log messages, so they work best with small, text-based files. Large files or binary data might get truncated or corrupted, so this is a workaround, not a perfect fit.

用rsyslog的步骤 #

1. 配置HDFS节点的rsyslog客户端 #

First, set up rsyslog on your HDFS node to forward messages to the remote server:

• Create or edit a config file (like /etc/rsyslog.d/hdfs-transfer.conf) with:

  # Forward all messages from the local7 facility to the remote server via UDP
  local7.* @your-remote-server-ip:514
  

• Restart rsyslog to apply changes:

  sudo systemctl restart rsyslog
  

2. 配置远程服务器的rsyslog接收端 #

On the remote server, configure rsyslog to listen for incoming messages and write them to a local file:

• Edit the main rsyslog config (e.g., /etc/rsyslog.conf) to enable UDP reception and route logs to your target file:

  # Load the UDP input module
  module(load="imudp")
  input(type="imudp" port="514")
  
  # Write all local7 messages to your target file
  local7.* /path/to/your/target/local/file
  

• Restart rsyslog:

  sudo systemctl restart rsyslog
  

3. 传输HDFS文件 #

Now pipe the HDFS file content into the logger command (which sends it to rsyslog):

hadoop fs -cat /path/to/your/hdfs/file | logger -p local7.info

The remote server's rsyslog will capture the content and write it to the file you specified.

用syslog-ng的步骤 #

The logic is similar, just config syntax differs:

1. HDFS节点的syslog-ng客户端 #

• Create a config snippet (e.g., /etc/syslog-ng/conf.d/hdfs-transfer.conf):

  source s_hdfs_pipe { pipe("/tmp/hdfs_transfer_pipe"); };
  destination d_remote_server { udp("your-remote-server-ip" port(514)); };
  log { source(s_hdfs_pipe); destination(d_remote_server); };
  

• Restart syslog-ng:

  sudo systemctl restart syslog-ng
  

• Pipe the HDFS file into the pipe:

  hadoop fs -cat /path/to/your/hdfs/file > /tmp/hdfs_transfer_pipe
  

2. 远程服务器的syslog-ng接收端 #

• Add this to syslog-ng config:

  source s_remote { udp(port(514)); };
  destination d_local_file { file("/path/to/your/target/file"); };
  log { source(s_remote); destination(d_local_file); };
  

• Restart syslog-ng to apply changes.

Since syslog isn't built for file transfer, these tools are more reliable and efficient, especially for large or binary files:

1. SCP/SFTP（最直接通用） #

If you can set up SSH key-based authentication between your HDFS node and remote server (for passwordless access), this is the simplest method:

• Option 1: Use a temporary local file

  # Download HDFS file to local temp
  hadoop fs -get /path/to/hdfs/file /tmp/hdfs_temp_file
  # SCP to remote server
  scp /tmp/hdfs_temp_file your-remote-user@your-remote-server:/path/to/target/dir/
  # Clean up temp file
  rm /tmp/hdfs_temp_file
  

• Option 2: Pipe directly (no temp file)

  hadoop fs -cat /path/to/hdfs/file | ssh your-remote-user@your-remote-server "cat > /path/to/target/file"
  

This works for all file types and handles large files well.

2. Rsync（适合增量传输） #

Rsync is great if you need to transfer files repeatedly—only the changed parts get sent, saving bandwidth:

• Pipe method:

  hadoop fs -cat /path/to/hdfs/file | ssh your-remote-user@your-remote-server "rsync --inplace - /path/to/target/file"
  

• Temporary file method (better for very large files):

  hadoop fs -get /path/to/hdfs/file /tmp/hdfs_temp_file
  rsync -avz /tmp/hdfs_temp_file your-remote-user@your-remote-server:/path/to/target/dir/
  rm /tmp/hdfs_temp_file
  

3. HDFS DistCp（如果远程服务器支持HDFS协议） #

If you can set up a lightweight HDFS client or a mini HDFS cluster on the remote server, you can use Hadoop's built-in distcp tool to directly copy between HDFS instances:

hadoop distcp hdfs://your-namenode-ip:9000/path/to/hdfs/file hdfs://remote-server-ip:9000/path/to/target/dir/

Note: This only works if the remote server can speak HDFS protocol.

Final Note #

If you're dealing with small text files, rsyslog/syslog-ng can get the job done. But for most real-world file transfer needs (especially large/binary files), SCP/SFTP or Rsync are far better choices—they're purpose-built for this task and avoid the limitations of syslog systems.

内容的提问来源于stack exchange，提问作者raghu