在AWS EMR中读取S3存储桶CSV文件无响应问题排查

Let’s walk through the likely issues and fixes for your scenario where Spark on EMR hangs without errors when reading S3 CSV files—even though your local setup works perfectly.

1. Verify S3 URI Format & Path Validity #

First up, double-check your S3 path syntax. While s3:// works on EMR, older versions like 5.21.0 sometimes have better compatibility with s3a:// (the Hadoop-native S3 client). Try modifying your read command to use this format:

df = spark.read.option("delimiter", ",").csv("s3a://{0}/{1}/*.csv".format(bucket_name, power_prod_key), header=True)

Also confirm:

• The bucket_name and power_prod_key are correctly spelled (S3 object keys are case-sensitive, while bucket names must be lowercase).
• The target files actually exist in the specified path—use the AWS S3 console to verify exact file paths, including any nested subfolders.

2. Validate IAM Role Permissions #

Even if you added S3 read/write permissions, ensure your roles have the exact necessary actions for listing and reading objects:

• For EMR_EC2_DefaultRole (the role attached to your cluster’s EC2 instances), add these permissions if missing:

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": [
                  "s3:GetObject",
                  "s3:ListBucket"
              ],
              "Resource": [
                  "arn:aws:s3:::powercaster-bct",
                  "arn:aws:s3:::powercaster-bct/*"
              ]
          }
      ]
  }
  

• s3:ListBucket is critical when using wildcards like *.csv—without it, Spark can’t enumerate the files in your bucket/prefix, leading to silent hangs.

3. Check Network & VPC Configuration #

If your EMR cluster is in a private subnet, it needs a S3 Gateway VPC Endpoint to access S3 directly (no internet routing required). Here’s how to confirm:

1. Go to the AWS VPC console, navigate to your subnet subnet-3482b47e.
2. Check if a Gateway Endpoint for S3 is attached to the subnet. If not, create one and associate it with your subnet and route table.
3. Ensure your EMR security groups (sg-05c284d83c1307807 and sg-01cd4e90f09dff3ad) allow outbound traffic to S3 (or the VPC endpoint).

4. Inspect Hidden Logs for Clues #

Since no exceptions are thrown, the issue might be buried in Spark’s detailed logs:

1. Go to the EMR console, select your cluster, and navigate to the Steps tab.
2. Click on your Spark application step, then view the Driver and Executor logs stored in your S3 log bucket (s3n://aws-logs-597071303168-us-east-1/elasticmapreduce/).
3. Look for entries related to S3 connections—common hidden issues include:
   • Permission denied errors (not always surfaced in your code’s output).
   • Timeouts when trying to reach S3 (indicates network misconfiguration).
   • Corrupted CSV files causing silent hangs (test with a small, known-good CSV file to rule this out).

5. Update EMR Version (Optional but Recommended) #

EMR 5.21.0 is quite outdated (released in 2019). It may have bugs in the S3 client or Spark integration that have been fixed in newer versions. Try launching a cluster with a more recent 5.x release (like emr-5.36.0) or 6.x release to see if the issue resolves itself.

6. Enable Debug Logging in Spark #

Add debug logging to your code to get granular visibility into what’s happening during the read operation:

spark.sparkContext.setLogLevel("DEBUG")
# Force an action to trigger the read (Spark is lazy-execution)
df = spark.read.option("delimiter", ",").csv("s3://{0}/{1}/*.csv".format(bucket_name, power_prod_key), header=True)
df.show()

The debug logs will show you exactly when Spark starts connecting to S3, how many files it’s trying to read, and any underlying issues that aren’t being thrown as exceptions.

内容的提问来源于stack exchange，提问作者César Bouyssi