使用Distcp从Hortonworks集群复制分区Parquet文件到Azure Blob失败求助

Hey there, let's dig into why your DistCP command is failing even though basic HDFS-to-Azure Blob operations work. Since you've confirmed the credential provider and Blob access is valid (via hadoop fs commands), we can rule out core authentication issues. Here are the most likely culprits and fixes:

1. Incomplete or Incorrect DistCP Command Syntax #

You mentioned your command was cut off (-skipcr...). First, make sure you're using the full, valid syntax—especially if you intended to skip checksum checks. The proper flag is -skipcrccheck, and you must explicitly specify both source and target paths. For partitioned Parquet files, target the root of the partitioned directory (not individual files) to preserve the hierarchical structure.

Example of a complete, valid command:

hadoop distcp -D hadoop.security.credential.provider.path=localjceks://file/tmp/azureb.jceks \
  -skipcrccheck \
  hdfs://<namenode-host>/path/to/sqoop/partitioned-parquet-root \
  wasbs://<container-name>@<storage-account>.blob.core.windows.net/target-directory

2. Partitioned Directory Structure Compatibility #

Sqoop generates partitioned Parquet files with a specific layout (e.g., year=2024/month=06/day=15). If you're using an older Azure Blob Storage connector with HDP 2.7.3, there might be bugs preserving this hierarchical structure or partition metadata.

• Verify you're using a connector version compatible with HDP 2.7.3—Microsoft recommends version 3.0.0+ for Hadoop 2.x, but cross-check HDP's official compatibility matrix to be safe.
• Add the -update flag if re-running the command to avoid overwriting existing files, which can trigger unexpected failures:

  hadoop distcp -D hadoop.security.credential.provider.path=localjceks://file/tmp/azureb.jceks \
    -skipcrccheck -update \
    hdfs://source-path \
    wasbs://target-path
  

3. Parquet File Permissions or Corruption #

Even if files exist in HDFS, Sqoop might have generated Parquet files with incorrect permissions or minor corruption that doesn't break read operations but disrupts DistCP.

• Check permissions for source files and directories:

  hdfs dfs -ls -R hdfs://<namenode-host>/path/to/sqoop/parquet-files
  

  Ensure the user running DistCP has read access to all files and nested directories.
• Validate a sample Parquet file to rule out corruption:

  parquet-tools meta hdfs://<namenode-host>/path/to/sqoop/parquet-files/year=2024/sample-file.parquet
  

  If validation fails, re-run your Sqoop job to regenerate clean files.

4. Analyze DistCP's YARN Logs for Exact Errors #

DistCP runs as a MapReduce job in Hadoop 2.x, so the most critical step is checking the detailed YARN application logs to get the exact failure reason.

• Find the failed DistCP job's application ID using:

  yarn application -list -appStates FAILED
  

• Retrieve the logs with:

  yarn logs -applicationId <your-app-id>
  

  Look for keywords like Permission denied, File not found, Corrupt file, or Azure-specific errors (e.g., StorageException) to pinpoint the root cause.

5. Verify Azure Blob Connector Configuration #

Double-check your core-site.xml for correct Azure Blob settings—even basic hadoop fs commands work, DistCP might require additional properties:

Ensure these properties are set (adjust values to match your setup):

<property>
  <name>fs.azure.account.key.<storage-account>.blob.core.windows.net</name>
  <value>${your-storage-key}</value>
</property>
<property>
  <name>fs.azure.blob.storage.enable.append.support</name>
  <value>true</value>
</property>

If using a credential provider, confirm the key alias in your jceks file matches the one referenced in core-site.xml.

内容的提问来源于stack exchange，提问作者johovic