在Hortonworks中使用Pywebhdfs时遇PyWebHdfsException错误求助

Let’s break down what’s going wrong here and fix it step by step:

Core Issue: Service Mix-Up #

You started the HBase REST Server on port 8000, but PyWebHdfsClient is built to communicate with HDFS's WebHDFS service—these are two entirely separate services with incompatible APIs. That’s exactly why you’re hitting the PyWebHdfsException: the client is sending HDFS-specific requests to an HBase service that has no idea how to process them.

Step 1: Stop the Misconfigured HBase REST Service #

First, shut down the HBase REST server you started on port 8000 to avoid port conflicts later:

/usr/hdp/current/hbase-master/bin/hbase-daemon.sh stop rest

Step 2: Identify the Correct WebHDFS Port #

In Hortonworks Data Platform (HDP), WebHDFS uses a default port that varies by version:

• For pre-3.x HDP versions: Default port is 50070
• For HDP 3.x and newer: Default port is 9870

To confirm the exact port for your cluster:

• If you use Ambari for cluster management, check the HDFS service configuration under "Advanced hdfs-site" for the dfs.namenode.http-address property.
• Or test connectivity directly with a curl command (swap the port with your version’s default):

  curl http://192.168.56.101:50070/webhdfs/v1/?op=GETHOMEDIRECTORY
  

  A valid JSON response means you’ve got the right port.

Step 3: Fix Your Python Code #

Update your code to use the correct WebHDFS port, and add a quick check to ensure the target directory exists (to avoid path/permission issues):

from pywebhdfs.webhdfs import PyWebHdfsClient
import os

# Replace with your actual WebHDFS port (50070 or 9870)
hdfs = PyWebHdfsClient(host='192.168.56.101', port='50070', user_name='maria_dev')
target_dir = 'user/maria_dev'
my_file = os.path.join(target_dir, 'creating.txt')

# Ensure the parent directory exists before creating the file
try:
    hdfs.get_file_dir_status(target_dir)
except Exception:
    hdfs.make_dir(target_dir)

# Now create the file
hdfs.create_file(my_file, 'My name is awesome')

Step 4: Additional Troubleshooting Checks #

If you still run into problems after the above steps:

• Firewall/Network: Verify the client machine can reach the namenode’s WebHDFS port (no firewall rules blocking traffic on 50070/9870).
• Permissions: Make sure the maria_dev user has write access to /user/maria_dev on HDFS. Set this via the command line if needed:

  hdfs dfs -mkdir -p /user/maria_dev
  hdfs dfs -chown maria_dev:maria_dev /user/maria_dev
  

• HDFS Logs: Check the namenode logs (typically at /var/log/hadoop/hdfs/) for connection or permission errors that can give more context.

内容的提问来源于stack exchange，提问作者Anwesh Mohapatra