本地Java远程创建HDFS目录：core-site与hdfs-site.xml配置疑问

Hey there! Let's walk through your HDFS Java code question, including clarifying what core-site.xml and hdfs-site.xml do, where to put them, and fixing a key issue in your current code.

1. What These Configuration Files Actually Do #

These two files are the backbone of Hadoop's configuration—they tell your client code (and the Hadoop cluster itself) how to behave:

• core-site.xml: This is Hadoop's global core config file. It defines foundational settings that apply across all Hadoop components, like:
  • fs.defaultFS: The default address for your HDFS cluster (this is the RPC endpoint for the NameNode, not the web UI)
  • hadoop.tmp.dir: The base temporary directory where Hadoop stores transient data (like NameNode metadata backups or DataNode temporary blocks)
• hdfs-site.xml: This file is HDFS-specific, fine-tuning how the distributed file system operates. Common settings here include:
  • dfs.replication: The default number of replicas for files stored in HDFS
  • dfs.namenode.name.dir: The path where the NameNode stores its critical metadata
  • dfs.datanode.data.dir: The path(s) where DataNodes store actual file blocks

2. Where to Put These Files for Your Java Client #

You have two straightforward options to make these configs available to your code:

• Package them in your project's classpath:
  If you're using a build tool like Maven or Gradle, drop the files into your project's src/main/resources directory. For non-build-tool projects, place them directly in the root of your classpath. Hadoop's Configuration object will automatically load these files on initialization, so you won't need to manually set fs.defaultFS in code.
• Load them manually via code:
  If you don't want to bundle the files with your project, you can explicitly specify their paths in your code:

  Configuration obj = new Configuration();
  obj.addResource(new Path("/absolute/path/to/your/core-site.xml"));
  obj.addResource(new Path("/absolute/path/to/your/hdfs-site.xml"));
  

  Just make sure the path is absolute and your application has read access to those files.

3. Fixing Your HDFS Directory Creation Code #

I noticed a critical issue in your current code: you're using http://datlpdsnn01.pds.in.****.com:50070 as your fs.defaultFS value. That's the NameNode web UI port, not the RPC port that Hadoop clients use to connect and perform operations like creating directories.

The standard RPC ports for NameNode are 8020 (Hadoop 2.x+) or 9000 (older versions). Update that line to use the HDFS protocol and correct port:

obj.set("fs.defaultFS", "hdfs://datlpdsnn01.pds.in.****.com:8020/");

Also, it's a good practice to use try-with-resources to automatically close the FileSystem object (avoids resource leaks) and check if the directory already exists before creating it:

public class HadoopCall { 
    public void demomkdir(String dir) throws IOException { 
        Configuration obj = new Configuration(); 
        obj.set("fs.defaultFS", "hdfs://datlpdsnn01.pds.in.****.com:8020/"); 
        
        // Try-with-resources auto-closes FileSystem when done
        try (FileSystem fs = FileSystem.get(obj)) {
            Path pth = new Path(dir); 
            if (!fs.exists(pth)) {
                fs.mkdirs(pth); 
                System.out.println("Directory created successfully: " + dir); 
            } else {
                System.out.println("Directory already exists: " + dir);
            }
        }
    } 

    public static void main(String[] args) throws IOException { 
        HadoopCall oj = new HadoopCall(); 
        oj.demomkdir("user/*******/javacodemkdir"); 
    } 
}

内容的提问来源于stack exchange，提问作者Abhishek Dutta