Mac Mojave下Hadoop 3.0.3部署时Secondary Namenode连接超时求助

Hey there, let's tackle these two issues you're running into with Hadoop 3.0.3 on Mac Mojave 10.14.6. I'll walk you through step-by-step fixes for both the SSH timeout and the native library warning.

Fixing the Secondary Namenode SSH Timeout Error #

The error ssh: connect to host xp port 22: Operation timed out means Hadoop can't establish an SSH connection to your machine's hostname xp. Here's how to resolve this:

1. Verify hostname resolution

   • First, check your actual machine hostname by running:

     hostname
     

   • Open your /etc/hosts file with a text editor using sudo:

     sudo nano /etc/hosts
     

   • Make sure you have entries mapping 127.0.0.1 and ::1 to your hostname xp, like this:

     127.0.0.1   localhost xp
     ::1         localhost xp
     

   • Save and close the file (for nano: press Ctrl+O, hit Enter, then Ctrl+X).

2. Confirm passwordless SSH is fully working

   • Even if you created keys, incorrect permissions can break passwordless login. Run these commands to set the right permissions:

     chmod 700 ~/.ssh
     chmod 600 ~/.ssh/authorized_keys
     

   • Test the connection directly by running:

     ssh xp
     

   • If you still get prompted for a password, double-check that your public key (~/.ssh/id_rsa.pub) was correctly appended to ~/.ssh/authorized_keys. Re-add it with:

     cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
     

3. Verify SSH service is running

   • Check if remote login is enabled:

     sudo systemsetup -getremotelogin
     

   • If it says Off, turn it on with:

     sudo systemsetup -setremotelogin On
     

   • You can also check if the SSH daemon is running:

     launchctl list | grep sshd
     

   • If it's not listed, start it manually:

     sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist
     

4. Check Hadoop configuration files

   • Open hdfs-site.xml in your Hadoop configuration directory (usually $HADOOP_HOME/etc/hadoop).
   • Verify the property dfs.namenode.secondary.http-address is set to xp:50090 (matches your hostname). If it's set to something else, update it to use xp or localhost.
   • Also, ensure core-site.xml has the correct fs.defaultFS value, like hdfs://xp:9000.

Addressing the NativeCodeLoader Warning #

The warning Unable to load native-hadoop library for your platform... is extremely common on Mac systems. Here's why and what to do:

• Hadoop's pre-built native libraries are compiled for Linux, so they don't work natively on Mac Mojave.
• Good news: This warning doesn't affect the core functionality of Hadoop. You can safely ignore it unless you're relying on performance-intensive features that require native libraries.
• If you really want to fix it, you'd need to compile Hadoop from source on your Mac to build compatible native libraries. But this is optional and adds unnecessary complexity for most use cases.

Once you've worked through the SSH fixes, re-run hdfs namenode -format and start Hadoop again. That should resolve the timeout error.

内容的提问来源于stack exchange，提问作者XavierBoa