Hive Streaming对接Azure Data Lake Store遇FileSystem异常问题

Looks like you're hitting a common issue where Hadoop can't locate the Azure Data Lake (ADL) FileSystem implementation even though you have the required JARs in your classpath. Let's break down the fixes step by step:

1. Add ADL FileSystem Configurations to Hadoop's core-site.xml #

Hadoop needs explicit configuration to map the adl:// scheme to its corresponding FileSystem class. Locate your cluster's core-site.xml (usually in /etc/hadoop/conf on HDInsight head nodes) and add these properties:

<property>
    <name>fs.adl.impl</name>
    <value>org.apache.hadoop.fs.adl.AdlFileSystem</value>
</property>
<property>
    <name>fs.AbstractFileSystem.adl.impl</name>
    <value>org.apache.hadoop.fs.adl.AdlAbstractFileSystem</value>
</property>

Without these entries, Hadoop won't know which class to use for handling ADL paths, even if the JARs are present.

2. Verify Dependency Version Compatibility #

Your current setup uses hadoop-azure-datalake-3.1.0.jar, but HDInsight clusters typically use specific Hadoop versions (many older clusters run Hadoop 2.7.x or 2.9.x). Mismatched versions can cause class loading failures or compatibility issues:

• Check your HDInsight cluster's Hadoop version by running hadoop version on the head node
• Replace the hadoop-azure-datalake JAR with the version that matches your cluster's Hadoop release. For example, if your cluster uses Hadoop 2.7.3, use hadoop-azure-datalake-2.7.3.jar instead of 3.1.0.

3. Ensure Play App's Classpath Includes the JARs Correctly #

If you're running the Play app directly on the head node, make sure the ADL JARs are properly included in the app's runtime classpath:

• If using SBT, confirm the dependencies are marked as compile or runtime (not provided) in your build.sbt:

  libraryDependencies += "org.apache.hadoop" % "hadoop-azure-datalake" % "your-compatible-version"
  libraryDependencies += "com.microsoft.azure" % "azure-data-lake-store-sdk" % "2.2.5"
  

• When launching the app, explicitly include the JARs in the classpath if needed (though SBT should handle this for packaged apps).

4. Manually Register the ADL FileSystem in Code (As a Last Resort) #

If the configuration changes don't take effect, you can force Hadoop to load the ADL FileSystem class by adding this initialization code before you start using Hive Streaming:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;

// ...

try {
    // Load the ADL FileSystem class explicitly
    Class.forName("org.apache.hadoop.fs.adl.AdlFileSystem");
    Configuration conf = new Configuration();
    // Register the scheme with the FileSystem
    FileSystem.getFileSystemClass("adl", conf);
} catch (ClassNotFoundException | IOException e) {
    e.printStackTrace();
    // Handle initialization failure appropriately
}

This bypasses any classpath or configuration loading issues by directly registering the implementation.

5. Double-Check ADL Path Permissions #

While the error points to a FileSystem issue, it's worth confirming the user running your Play app has read/write permissions on the ADL path adl://home/hive/warehouse/data/ingest_date=2018-05-07. You can test this with the adl CLI on the head node to ensure access works outside your app.

内容的提问来源于stack exchange，提问作者Thomas Nys