PySpark新手求助：使用PySpark读取JSON数据失败，附代码求指导

Hey there! Let's walk through why your PySpark JSON reading might be failing, based on the code you shared. Here are the most common culprits and fixes to try:

1. Incorrect JSON File Format #

PySpark is picky about JSON structure—it expects either:

• Line-delimited JSON: Each line is a standalone JSON object (like {"name": "Alice"} on one line, {"name": "Bob"} on the next)
• A single valid JSON array (like [{"name": "Alice"}, {"name": "Bob"}])

If your file is pretty-printed with multiple objects spread across lines (not line-delimited), Spark will throw a parse error.

Fix: Add the multiLine=True option to your read command:

df = spark.read.option("multiLine", True).json(fileName)

2. Local vs. HDFS Path Confusion #

Your code uses os.path.isfile(filename) to check if the file exists—but this only works for local filesystem paths. If you're trying to read a file from HDFS (e.g., hdfs:///user/data/file.json), os.path won't recognize it, and you'll get a false "File does not exist" message.

Fix:

• For local files: Use the full absolute path (e.g., /home/you/data/file.json) if your script isn't in the same directory as the file.
• For HDFS files: Skip the os.path check (or use HDFS commands like hdfs dfs -test -f filename to verify existence before running the script), and pass the full HDFS path to spark.read.json().

3. Missing Spark Session Configurations #

If you're running in a cluster environment, your Spark session might be missing key configurations to access files (like HDFS credentials or cluster settings).

Fix: Ensure your Spark session is set up properly—here's a more robust example:

spark = SparkSession \
    .builder \
    .appName("JSONReaderApp") \
    .getOrCreate()

If you're using a specific cluster manager (like YARN), add the appropriate configs (e.g., .master("yarn")).

4. Permission Issues #

Even if the file exists, Spark might not have permission to read it. This is common with local files owned by another user, or HDFS files with restricted access.

Fix:

• For local files: Check the file permissions with ls -l filename and ensure the user running your script has read access.
• For HDFS files: Use hdfs dfs -ls filename to verify permissions, and adjust them if needed (e.g., hdfs dfs -chmod +r filename).

One more thing—if you can share the exact error message you're seeing (like a JsonParseException or FileNotFoundException), that would help pinpoint the issue even faster!

内容的提问来源于stack exchange，提问作者Prashant Patel