使用Spark 2.2无法访问S3数据的问题求助

Hey there! Let's tackle this S3 data loading issue with your p7hb/docker-spark setup step by step. I’ve worked through similar containerized Spark + S3 scenarios before, so here’s what to check and fix:

Spark doesn’t ship with AWS SDK dependencies out of the box, and this is the most common culprit for failed S3 connections. Let’s confirm they’re present in your container:

• SSH into your running container or run this command directly:

  docker exec -it <your-spark-container-name> ls $SPARK_HOME/jars | grep -E "(hadoop-aws|aws-java-sdk)"
  

• If you don’t see matching jars, you have two options:
  • Mount jars at container startup: Download compatible versions (match your Spark/Hadoop version in the container—e.g., Spark 3.3.0 pairs with hadoop-aws:3.3.4 and aws-java-sdk-bundle:1.12.500) and mount them to the Spark jars directory:

    docker run -v /path/to/your/local/jars/:/opt/spark/jars/ p7hb/docker-spark
    

  • Specify jars in your Spark job: When submitting a job or running code in Zeppelin, add the dependencies via --jars (for CLI) or Zeppelin interpreter settings (more on that later).

Your Spark container needs valid AWS credentials to access S3. Pick one of these reliable methods:

• Inject environment variables at startup:

  docker run -e AWS_ACCESS_KEY_ID="your-access-key" -e AWS_SECRET_ACCESS_KEY="your-secret-key" p7hb/docker-spark
  

• Mount your local AWS credentials file:

  docker run -v ~/.aws/credentials:/root/.aws/credentials p7hb/docker-spark
  

• Set credentials directly in your Spark code:

  from pyspark.sql import SparkSession
  
  spark = SparkSession.builder.appName("S3Test").getOrCreate()
  hadoop_conf = spark.sparkContext._jsc.hadoopConfiguration()
  hadoop_conf.set("fs.s3a.access.key", "your-access-key")
  hadoop_conf.set("fs.s3a.secret.key", "your-secret-key")
  

  Pro tip: Use the s3a:// protocol instead of s3://—it’s Hadoop’s modern, more stable S3 implementation.

Depending on your S3 region or setup, you may need to add these extra settings:

• For AWS China regions or S3-compatible storage:

  hadoop_conf.set("fs.s3a.endpoint", "s3.cn-north-1.amazonaws.com.cn") # Replace with your region's endpoint
  

• Ensure the correct filesystem implementation is used:

  hadoop_conf.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
  

Before diving into Spark, rule out network/permission issues with a simple CLI test:

• Install the AWS CLI in your container (if missing):

  docker exec -it <your-spark-container-name> apt-get update && apt-get install -y awscli
  

• Run a list command to confirm access:

  aws s3 ls s3://your-bucket-name
  

If this fails, you know the problem is with network, credentials, or bucket permissions—not Spark.

If you’re running code in Zeppelin, make sure the Spark interpreter is set up correctly:

• Go to Zeppelin’s Interpreter settings, find the Spark interpreter.
• Add these Maven dependencies to the Dependencies section (adjust versions to match your setup):

  org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.500
  

• Under Environment Variables, add your AWS credentials:

  AWS_ACCESS_KEY_ID=your-access-key
  AWS_SECRET_ACCESS_KEY=your-secret-key
  

Here’s a minimal Python snippet to validate everything works:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("S3LoadTest").getOrCreate()

# Uncomment if you didn't set credentials via env/credentials file
# hadoop_conf = spark.sparkContext._jsc.hadoopConfiguration()
# hadoop_conf.set("fs.s3a.access.key", "your-access-key")
# hadoop_conf.set("fs.s3a.secret.key", "your-secret-key")

# Load a sample CSV from S3
df = spark.read.csv("s3a://your-bucket/path/to/data.csv", header=True, inferSchema=True)
df.show(5)

内容的提问来源于stack exchange，提问作者user1129682