PySpark 4.0.0中Spark Connect禁用回退仍自动降级至本地模式咨询

This is a confirmed bug in Spark 4.0.0 #

The behavior you're observing—where spark.connect.disable.fallback=true fails to prevent fallback to local mode when the Spark Connect server is unreachable—is a known issue in Spark 4.0.0. The configuration flag wasn't properly validated during SparkSession initialization, leading to silent fallback even when explicitly disabled.

Fix Availability #

This bug was resolved in Spark 4.0.1 and all subsequent minor (4.0.x) and major releases. Upgrading your PySpark installation to Spark 4.0.1 or newer will enforce the spark.connect.disable.fallback=true setting as intended: if the Connect server is unreachable, session initialization will fail immediately instead of falling back to local mode.

Temporary Workaround (if upgrade isn't feasible immediately) #

If you can't upgrade right now, add an explicit check after session creation to verify you're connected to the remote server instead of running locally:

spark1 = SparkSession.builder \
    .config("spark.api.mode", "connect") \
    .config("spark.connect.server.address", "sc://192.x.x.x:7077") \
    .config("spark.connect.disable.fallback", "true") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
    .getOrCreate()

# Validate session isn't running in local mode
if spark1.sparkContext._jsc.sc().isLocal():
    raise RuntimeError("Failed to connect to Spark Connect server; fallback to local mode detected despite disable.fallback=true")

# Proceed with operations
df = spark1.createDataFrame([{"name": "Alice", "age": 30}, {"name": "Bob", "age": 25}])
df.show(truncate=False)

This check will throw an error if unintended fallback occurs, aligning with your expected behavior until you can upgrade.

内容的提问来源于stack exchange，提问作者awesome_sangram