在Google Colab中运行Spark遇多类错误，求解决方案

Case 1: FileNotFoundError: [Errno 2] No such file or directory: '/content/spark-3.0.2-bin-hadoop2.7/./bin/spark-submit' #

错误原因 #

直接安装pyspark后，Colab环境无法正确定位Spark执行文件的路径，导致初始化失败。

解决方案 #

通过findspark工具自动定位Spark路径，同时确保Java环境已安装：

!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!pip install pyspark findspark

import findspark
findspark.init()
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Basics").getOrCreate()
spark

Case 2: Py4JError: org.apache.spark.api.python.PythonUtils.getPythonAuthSocketTimeout does not exist in the JVM #

错误原因 #

手动下载的Spark版本（3.0.0）与pip安装的pyspark版本不兼容，导致JVM端与Python端API不匹配。

解决方案 #

统一Spark和pyspark版本，无需手动下载Spark，直接通过pip安装对应版本的pyspark：

!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!pip install pyspark==3.0.0 findspark

import findspark
findspark.init()
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Basics").getOrCreate()
spark

Case 3: ValueError: Cannot run multiple SparkContexts at once; existing SparkContext(...) created by init at :5 #

错误原因 #

SparkSession初始化时会自动创建对应的SparkContext，手动创建SparkContext会导致冲突。

解决方案 #

删除手动创建SparkContext的代码，通过SparkSession.builder配置参数：

!pip install pyspark findspark
!apt install openjdk-8-jdk-headless -qq

import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
import findspark
findspark.init()

from pyspark.sql import SparkSession
from pyspark.sql.types import *
from sklearn.datasets import load_breast_cancer
import pandas as pd
from pyspark.ml.linalg import Vectors

# 通过builder配置参数，无需手动创建SparkContext
spark = SparkSession.builder.appName("Basics").config("spark.ui.port", "4050").getOrCreate()

breast_cancer = load_breast_cancer()
pd_df = pd.DataFrame(breast_cancer.data, columns=breast_cancer.feature_names)
df = spark.createDataFrame(pd_df)

def set_df_columns_nullable(spark, df, column_list, nullable=False):
    # 安全修改Schema：创建新的StructType
    new_schema = StructType([
        StructField(f.name, f.dataType, nullable if f.name in column_list else f.nullable)
        for f in df.schema.fields
    ])
    df_mod = spark.createDataFrame(df.rdd, new_schema)
    return df_mod

df = set_df_columns_nullable(spark, df, df.columns)
df = df.withColumn('features', array(df.columns))
vectors = df.rdd.map(lambda row: Vectors.dense(row.features))

df.printSchema()
features = spark.createDataFrame(vectors.map(lambda x: (x,)), ["features"])
labels = pd.Series(breast_cancer.target)

Case 4: Py4JJavaError: An error occurred while calling o30.csv. #

错误原因 #

读取S3路径需要Hadoop AWS相关依赖包，Colab默认Spark环境未包含这些组件，无法识别s3a协议。

解决方案 #

在SparkSession初始化时添加Hadoop AWS依赖，如需访问私有存储需配置AWS凭证：

!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!pip install pyspark findspark

import findspark
findspark.init()
from pyspark.sql import SparkSession

# 添加与Spark版本匹配的Hadoop AWS依赖
spark = SparkSession.builder.appName("Basics")\
    .config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.2.0")\
    .getOrCreate()

# 访问私有S3时需配置凭证（取消注释并替换为实际密钥）
# spark.conf.set("spark.hadoop.fs.s3a.access.key", "YOUR_ACCESS_KEY")
# spark.conf.set("spark.hadoop.fs.s3a.secret.key", "YOUR_SECRET_KEY")

DF = spark.read.csv("s3a://raw-recipes-clean-upgrad/RAW_recipes_cleaned.csv", inferSchema=True, header=True)
DF.show(5)

内容的提问来源于stack exchange，提问作者tran su