Spark2以YARN为Master提交任务时出现“URL must be set”错误排查

Hey there, let's break down this confusing issue—you've explicitly set --master yarn and --deploy-mode cluster in your spark2-submit command, but Spark's still throwing that master URL error, and you suspect KuduContext/KuduSink is the culprit. Since this worked before, it's likely a configuration mismatch or a hardcoded setting that's messing things up in cluster mode.

Here are the key steps to diagnose and fix this:

1. Check for Hardcoded Master Settings in Your Code #

This is the most common culprit. When testing locally, it's easy to add something like setMaster("local[*]") to your SparkConf, and forget to remove it when moving to cluster mode. For example:

// ❌ Bad: Hardcoded master overrides yarn cluster settings
val conf = new SparkConf().setMaster("local[*]").setAppName("MyKuduApp")
val sc = new SparkContext(conf)

Fix: Delete any hardcoded setMaster calls entirely. In YARN cluster mode, Spark automatically picks up the master from your submit command—you don't need to set it in code.

2. Verify KuduContext Initialization #

KuduContext relies on the correct Spark configuration to work in cluster mode. If you're initializing it with a brand-new SparkConf instead of reusing the one from your existing SparkContext/SparkSession, it won't inherit the YARN master settings.

• ❌ Wrong way: Creating a standalone SparkConf for Kudu

val kuduConf = new SparkConf().set("kudu.master", "your-kudu-master:7051")
val kuduContext = new KuduContext("your-kudu-master:7051", kuduConf)

• ✅ Correct way: Reuse the existing SparkSession/SparkContext's configuration

val spark = SparkSession.builder().appName("MyKuduApp").getOrCreate()
// Use the conf from your active SparkContext
val kuduContext = new KuduContext("your-kudu-master:7051", spark.sparkContext.conf)

If you're using KuduSink with Streaming/Datasets, make sure it's tied to your existing SparkSession—don't create a separate context just for Kudu operations.

3. Rule Out Cloudera Cluster Configuration Conflicts #

Since this worked before, cluster config issues are less likely, but it's worth checking:

• Double-check your spark2-submit command for accidental --conf overrides—like --conf spark.master=local that would overwrite your --master yarn setting.
• Ensure you're using the official Cloudera spark2-submit script (not a custom one) to make sure it pulls the correct cluster defaults from spark-defaults.conf.

4. Dig Into the Full Exception Stack Trace #

You mentioned the error involves KuduSink—grab the full stack trace to see exactly which line is triggering the master URL error. If it's KuduSink trying to spin up a new SparkContext internally, that's the problem: you should always reuse the context from your running job instead of creating a new one in the sink.

For example, in a Streaming job, make sure your Kudu operations inside foreachRDD use the RDD's existing SparkContext, not a new one.

内容的提问来源于stack exchange，提问作者Alter