本地Hadoop访问AWS S3报Unable to load AWS credentials错误求助

Let's break down exactly what's going wrong here and how to fix it—your AWS CLI works, so the credentials themselves are valid, which narrows things down to Hadoop S3A configuration missteps.

Key Issues Causing the Error #

1. Incorrect Configuration Parameter Names #

You’re using fs.s3.access.key and fs.s3.secret.key in your command, but these belong to the old, deprecated S3 filesystem client. The S3A client (which you’re targeting with s3a://) uses parameters prefixed with fs.s3a. instead.

When you specify TemporaryAWSCredentialsProvider, it only looks for the fs.s3a.* versions of these parameters—so your current command isn’t actually passing credentials to the provider you’re trying to use.

2. Credential Provider Behavior #

The TemporaryAWSCredentialsProvider doesn’t read environment variables (like your exported AWS_ACCESS_KEY_ID). It strictly relies on the three explicit fs.s3a.* configuration parameters you pass. Your environment variables are ignored here because you’ve overridden the default provider chain.

3. Potential Command Order Misconfiguration #

The -libjars flag should be specified as a generic option for the hadoop fs command. Placing it early ensures Hadoop loads the required hadoop-aws.jar before processing S3A-specific commands, avoiding classpath issues.

Fixes to Try #

Option 1: Fix Parameters for Temporary Credentials Provider #

Update your command to use the correct fs.s3a.* parameter names, and ensure all three temporary credential fields are properly set:

hadoop fs -libjars <path to hadoop-aws.jar file> \
  -Dfs.s3a.aws.credentials.provider="org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider" \
  -Dfs.s3a.access.key="<Your Access Key>" \
  -Dfs.s3a.secret.key="<Your Secret Access Key>" \
  -Dfs.s3a.session.token="<Your Session Token>" \
  -ls s3a://<bucket_name>/

Double-check that your secret key and session token values are copied correctly (no typos or extra spaces!).

Option 2: Use Default Credential Chain (Leverage Environment Variables) #

If you want to use the environment variables you’ve already set, remove the explicit provider configuration. The DefaultAWSCredentialsProviderChain (the default if you don’t specify a provider) automatically reads environment variables, AWS credential files, and IAM roles (if running on EC2/EKS).

Your command simplifies to:

hadoop fs -libjars <path to hadoop-aws.jar file> \
  -ls s3a://<bucket_name>/

This will pick up the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN variables you exported earlier.

Bonus: Verify Hadoop-AWS Compatibility #

Make sure the version of hadoop-aws.jar you’re using matches your Hadoop cluster version. Mismatched versions can cause unexpected credential loading failures or other S3A client issues. For example, Hadoop 3.3.x requires hadoop-aws-3.3.x.jar.

内容的提问来源于stack exchange，提问作者madhu sudhan