请教createOrReplaceGlobalTempView与createOrReplaceTempView的差异及Spark应用与会话区别

Great question—let’s unpack this clearly, starting with the two view methods, then diving into the core distinctions between Spark Applications and SparkSessions that make their behaviors different.

First: The View Methods at a Glance #

• createOrReplaceTempView (Spark 2.0+): This creates a temporary view tied directly to the SparkSession that created it. Once that session is closed or garbage-collected, the view vanishes.
• createOrReplaceGlobalTempView (Spark 2.2+): This creates a global temporary view tied to the entire Spark Application. It sticks around until the entire application shuts down, and is accessible to all SparkSessions within that application.

To really get why these behave differently, we need to clarify what a Spark Application vs SparkSession actually is.

Spark Application vs SparkSession: Key Differences #

Think of a Spark Application as the "big container" for your entire Spark workload, and SparkSessions as "individual workspaces" inside that container. Here’s the breakdown:

1. Scope & Boundaries #

• Spark Application: It’s the full instance of your Spark job, starting when you run spark-submit, launch a Spark shell, or start a Spark application programmatically. It includes all the cluster resources allocated to your job (executors, memory, CPU cores) and runs until the job completes or you explicitly terminate it (like closing the shell).
• SparkSession: Introduced in Spark 2.0 as the unified entry point for interacting with Spark, a session represents a single, isolated interaction with the Spark cluster. One Spark Application can have multiple SparkSessions—for example, different users connecting to a long-running Spark application, or different modules in your code needing isolated workspaces.

2. Lifecycle #

• Spark Application: Its lifecycle is the entire duration of your workload. When the application ends, every resource tied to it (including global temporary views) is cleaned up automatically.
• SparkSession: Its lifecycle starts when you create it (via SparkSession.builder().getOrCreate()) and ends when you call stop() on it or it’s garbage-collected. When a session dies, any temporary views created with createOrReplaceTempView are deleted, but the application and other sessions keep running.

3. Resource & Data Sharing #

• Spark Application: Global temporary views are shared across all sessions in the application. To access them, you need to use the global_temp. prefix (e.g., spark.sql("SELECT * FROM global_temp.my_shared_view")). This is perfect when you need to share intermediate data between multiple sessions or users in the same application.
• SparkSession: Regular temporary views are private to the session that created them. No other session in the same application can see or access them—they’re isolated workspaces for specific tasks.

4. Practical Use Cases #

• Use createOrReplaceTempView when: You’re working within a single session (like a standalone script or a single analysis task) and don’t need to share the view with other parts of your application.
• Use createOrReplaceGlobalTempView when: You need to share data across multiple sessions (e.g., in a long-running Spark shell where you create multiple sessions, or a multi-user Spark application where different users need access to the same shared data).

Example to Illustrate the Behavior #

Let’s walk through a quick Scala example in the Spark shell to see this in action:

// Create the first SparkSession
val spark1 = SparkSession.builder().appName("MyTestApp").getOrCreate()
// Create a session-local temp view
spark1.range(1, 6).createOrReplaceTempView("local_numbers")
// Create a global temp view
spark1.range(10, 16).createOrReplaceGlobalTempView("global_numbers")

// Create a second, independent SparkSession in the same application
val spark2 = spark1.newSession()

// Try accessing the views from spark2
spark2.sql("SELECT * FROM global_temp.global_numbers").show() // Works! Returns 10-15
spark2.sql("SELECT * FROM local_numbers").show() // Fails—view doesn't exist in spark2

// Stop the first session
spark1.stop()

// spark2 is still running, and the global view is still accessible
spark2.sql("SELECT * FROM global_temp.global_numbers").show() // Still works!

// Only when we close the Spark shell (end the application) does the global view disappear

内容的提问来源于stack exchange，提问作者Neeraj Bhadani