Zeppelin中Spark2.0环境下SparkSession创建问题咨询

Hey there! Let's break down how to handle SparkSession creation when using Spark 2.0 in Zeppelin—this is straightforward once you know the ins and outs.

First: Zeppelin's Auto-Initialized SparkSession #

Spark 2.0 introduced SparkSession as the unified entry point for Spark SQL, and Zeppelin does the heavy lifting here by default. As soon as you fire up the Spark interpreter in Zeppelin, it automatically creates a SparkSession instance for you, accessible via the variable spark. You can start using it right away without writing any setup code!

To confirm it's working, run this quick test in a Zeppelin paragraph:

// Verify the session is active by checking Spark version
println(spark.version)

// Run a simple test query to confirm functionality
spark.sql("SELECT 'Hello from SparkSession!' AS greeting").show()

Creating a Custom SparkSession #

If you need a session with specific configurations (like a custom app name, adjusted resource limits, or unique config flags), use the getOrCreate() method. This ensures you don't accidentally create duplicate sessions (which would throw an error) and reuses the existing one if available. Here's a example:

import org.apache.spark.sql.SparkSession

// Build a custom session (or reuse the existing one)
val customSpark = SparkSession.builder()
  .appName("MyCustomZeppelinSession")
  .config("spark.sql.shuffle.partitions", "8") // Example custom config
  .config("spark.driver.memory", "2g") // Adjust based on your environment's resources
  .getOrCreate()

// Test the custom session with a sample read operation
customSpark.read.text("/path/to/your/sample-file.txt").show()

Important Tips #

• Skip builder().build() alone: Always use getOrCreate() because Zeppelin maintains an active interpreter context—directly building a new session will conflict with it.
• Leverage Zeppelin's interpreter settings: For global configurations that apply to all sessions, set Spark properties in the Zeppelin Spark interpreter settings (under the "Interpreter" tab) instead of defining them in code. This keeps your paragraphs clean and consistent.

内容的提问来源于stack exchange，提问作者vero