Great question! While DistCp doesn’t have native, out-of-the-box support for direct transfers to NFS or SMB shares (unlike other Hadoop clusters, local directories, or cloud blobs), there are two practical, widely-used approaches to achieve this.

Approach 1: Mount the Remote Share Locally on Cluster Nodes #

This is the most straightforward method—treat the NFS/SMB share as a local directory by mounting it on your Hadoop cluster nodes, then use DistCp to copy to that mounted path.

Step 1: Mount the Remote Share on All Relevant Nodes #

DistCp runs in parallel across your cluster, so you’ll need to mount the share on all DataNodes and the node where you’ll run the DistCp command (usually the NameNode or a gateway node).

For NFS Shares:

# Create a local mount point (e.g., /mnt/nfs-hadoop-share)
sudo mkdir -p /mnt/nfs-hadoop-share

# Mount the NFS share (replace placeholders with your server details)
sudo mount -t nfs <nfs-server-ip>:/path/to/remote/share /mnt/nfs-hadoop-share

# Optional: Make the mount persistent across reboots by adding to /etc/fstab
echo "<nfs-server-ip>:/path/to/remote/share /mnt/nfs-hadoop-share nfs defaults 0 0" | sudo tee -a /etc/fstab

For SMB Shares:

First install the required utilities, then mount the share:

# Install cifs-utils (Debian/Ubuntu)
sudo apt-get update && sudo apt-get install cifs-utils -y

# Or for RHEL/CentOS
sudo yum install cifs-utils -y

# Create a mount point
sudo mkdir -p /mnt/smb-hadoop-share

# Mount the SMB share (replace credentials and server details)
sudo mount -t cifs //<smb-server-hostname>/share-name /mnt/smb-hadoop-share -o username=<smb-username>,password=<smb-password>,uid=hdfs,gid=hdfs

# Optional: Persist in /etc/fstab
echo "//<smb-server-hostname>/share-name /mnt/smb-hadoop-share cifs username=<smb-username>,password=<smb-password>,uid=hdfs,gid=hdfs 0 0" | sudo tee -a /etc/fstab

Step 2: Adjust Permissions for Hadoop User #

Ensure the Hadoop runtime user (typically hdfs) has read/write access to the mounted directory. The uid=hdfs,gid=hdfs option in the SMB mount command handles this, but for NFS, you may need to tweak the share’s export settings on the NFS server or run:

sudo chown -R hdfs:hdfs /mnt/nfs-hadoop-share

Step 3: Run DistCp to Copy Data #

Treat the mounted share like a local directory in your DistCp command:

# Copy from HDFS to NFS share
hadoop distcp hdfs://<local-nn-host>:8020/path/to/source/data /mnt/nfs-hadoop-share/target-directory

# Copy from HDFS to SMB share
hadoop distcp hdfs://<local-nn-host>:8020/path/to/source/data /mnt/smb-hadoop-share/target-directory

You can use DistCp flags to optimize the transfer:

• -update: Only copy files that are new or modified (incremental transfer)
• -skipcrccheck: Skip CRC checks for faster transfers (use only if you trust the network)
• -m <num>: Set the number of parallel copy tasks (adjust based on your cluster and network capacity)

Example with incremental transfer:

hadoop distcp -update -m 20 hdfs://<local-nn-host>:8020/path/to/source/data /mnt/nfs-hadoop-share/target-directory

Approach 2: Use Hadoop’s SMBFileSystem (SMB Only) #

If you’re working with SMB shares and prefer not to mount them locally, Hadoop 3.x+ includes an SMBFileSystem implementation that lets you directly access SMB shares via DistCp.

Step 1: Configure Hadoop for SMB Access #

Add the following property to your core-site.xml file (located in $HADOOP_CONF_DIR):

<property>
  <name>fs.smb.impl</name>
  <value>org.apache.hadoop.fs.smb.SMBFileSystem</value>
</property>

Step 2: Run DistCp with the SMB Protocol #

Use the smb:// URI format to specify the SMB share target. Include your credentials directly in the URI (or use a credential provider for security):

hadoop distcp hdfs://<local-nn-host>:8020/path/to/source/data smb://<smb-username>:<smb-password>@<smb-server-hostname>/share-name/target-directory

Notes for SMBFileSystem: #

• Ensure your Hadoop classpath includes the necessary SMB-related JARs (usually included in the default Hadoop 3.x distribution).
• For better security, avoid hardcoding credentials in the command—use Hadoop’s credential provider API to store them securely.
• Performance may vary compared to the mount method; adjust parallelism (-m flag) based on your SMB server’s capacity.

Key Considerations #

• Cluster-wide Mounts: For Approach 1, ensure the share is mounted on every DataNode to avoid "file not found" errors during parallel transfers.
• Network Bandwidth: Both methods depend on the network link between your Hadoop cluster and the remote share—test with small datasets first to validate throughput.
• Error Handling: DistCp will log failures to a temporary directory on HDFS; check these logs if transfers fail to troubleshoot issues like permission denied or network drops.

内容的提问来源于stack exchange，提问作者Tamer Sherif