Spark读取5GB JSON文件报Java heap space错误的配置排查求助

Hey Jenny, let's work through this OutOfMemoryError issue together—reading large JSON files in Spark can be tricky at first, but adjusting your configs and understanding how Spark handles data will get you sorted.

First, Let's Flag Issues in Your Current Config #

Your existing settings have a few red flags that might be contributing to the OOM error:

1. spark.sql.shuffle.partitions = 8000
   This is way too high! This parameter controls the number of partitions during shuffle operations (like joins or aggregations). Even if you haven't gotten to that step yet, setting it this high wastes memory and processing power—each partition will be tiny, and Spark will spend more time managing partitions than processing data. Aim for a value that aligns with your cluster's capacity: start with 200-500 (a good rule of thumb is 2-4 partitions per CPU core across your cluster).

2. spark.memory.fraction = 0.8
   This sets the share of executor memory allocated to Spark's storage and execution tasks. 0.8 leaves only 20% of executor memory for user code, system processes, and unexpected spikes—this is tight and can easily trigger OOM. Stick with the default 0.6 (or 0.7 if you're confident you don't need extra headroom for user code).

3. spark.driver.memory = 12g & spark.executor.memory = 14g
   These values depend entirely on the actual memory available on your driver and executor nodes:

   • If you're running in local mode (your laptop/desktop), your driver and executor share the same JVM. If your machine has, say, 16GB of RAM, allocating 12GB to the driver leaves almost nothing for your OS and other apps—cut this to 8GB max.
   • If you're on a cluster, each executor node needs 2-4GB of RAM reserved for the OS and background processes. If your executor nodes have 16GB total RAM, set spark.executor.memory to 12GB instead of 14GB. Don't over-allocate—Spark can't use memory that doesn't exist!

Additional Configs to Add for Large JSON Reads #

Here are key settings to add or adjust to handle your 5GB JSON file:

• spark.executor.cores
  Default is 1, which means each executor only uses one CPU core. If you're allocating 12-14GB to each executor, you can safely set this to 4-8 (match the number of cores available on each executor node). More cores per executor improves parallelism and reduces the memory load per task.

• spark.driver.maxResultSize
  Default is 1GB. If you ever run operations like collect() to bring data to the driver (which you should avoid for large datasets!), this can trigger OOM. Increase it to 4-8GB to give the driver more headroom, but remember: prefer Spark SQL transformations over collecting data to the driver whenever possible.

• spark.sql.json.parser.columnNameOfCorruptRecord
  Add this to capture malformed JSON records instead of letting them crash your job. For example:

  .config("spark.sql.json.parser.columnNameOfCorruptRecord", "_corrupt_record")
  

  This will store bad records in a dedicated column, so you can inspect them later without failing the entire read.

Optimize the JSON Read Itself #

Beyond configs, how you read the JSON matters:

• Check your JSON format:

  • If your JSON is line-delimited (one JSON object per line), Spark can split the file across partitions automatically—this is ideal for large files.
  • If it's a multi-line JSON (one big JSON object spanning multiple lines), you need to add .option("multiLine", "true") to your read statement. However, multi-line JSON can't be split, so the entire file will be loaded into a single partition. If this is your case, consider splitting the file into smaller chunks first (e.g., using split on Unix) to avoid loading 5GB into one executor's memory.

• Test with a sample first:
  Instead of reading the entire 5GB file every time, test your configs with a sample:

  df = spark.read.json("example.json").sample(0.1)
  

  This lets you validate your settings without waiting for a full read.

Revised Example Config #

Here's how your config might look after adjustments (tweaked for a typical 16GB-node cluster or powerful desktop):

spark = SparkSession \
 .builder \
 .appName("Python Spark SQL basic example") \
 .master("local[*]")  # Add this if running locally—uses all available cores
 .config("spark.memory.fraction", 0.6) \
 .config("spark.executor.memory", "12g") \
 .config("spark.driver.memory", "8g") \
 .config("spark.executor.cores", 4) \
 .config("spark.sql.shuffle.partitions", 200) \
 .config("spark.driver.maxResultSize", "4g") \
 .config("spark.sql.json.parser.columnNameOfCorruptRecord", "_corrupt_record") \
 .getOrCreate()

Final Tips #

• Avoid collect() on large datasets—use write() to save results to storage instead, or work entirely with Spark SQL queries.
• Monitor Spark's UI (default port 4040) to check memory usage, partition sizes, and task performance—this will help you fine-tune configs further.

内容的提问来源于stack exchange，提问作者Jenny