Spark 2作业在Oozie工作流中执行失败：指定库目录不存在

Hey there! Let's figure out why your Spark job runs fine with spark-submit but throws that frustrating directory-missing error in Oozie. The java.lang.IllegalStateException about the missing assembly jars directory usually stems from how Oozie resolves Spark dependencies or configures the job environment. Here are the most common fixes:

1. Package Your Job as an Uber (Fat) Jar #

When you run with spark-submit locally, your machine probably has access to Spark's built assembly jars. But Oozie runs jobs on YARN containers, which don't have that local path available.

• Use tools like sbt-assembly or maven-shade-plugin to build a single "fat jar" that includes all your job's dependencies (exclude Spark core dependencies to avoid conflicts with the cluster's Spark installation).
• Upload this jar to HDFS and reference it in your Oozie workflow instead of relying on local Spark assembly paths.

2. Configure Spark Paths Correctly in Oozie Workflow #

Tell Oozie exactly where to find Spark's jars on the cluster. Update your workflow.xml spark-action with these settings:

<spark xmlns="uri:oozie:spark-action:0.2">
    <!-- Other required configs like job-tracker, name-node, class, jar -->
    <spark-opts>
        --spark-home /usr/lib/spark 
        --conf spark.yarn.jars=hdfs:///path/to/cluster/spark/jars/*.jar
    </spark-opts>
</spark>

• Replace /usr/lib/spark with your cluster's actual Spark installation path.
• The spark.yarn.jars config points to Spark's jars stored in HDFS, so YARN containers can pull them directly without relying on local directories.

3. Check YARN Container Permissions (Less Common but Worth Verifying) #

The error references a YARN local cache directory. Ensure:

• The user running Oozie (in your case, sshuser) has read/write permissions on YARN's local directories (configured via yarn.nodemanager.local-dirs in yarn-site.xml).
• The YARN local directories exist and aren't full or corrupted.

4. Confirm Cluster Spark Installation #

If you're running spark-submit in local mode but Oozie uses cluster mode, double-check that the cluster's Spark installation is properly built and configured. The SPARK_HOME environment variable should be set correctly on all worker nodes.

Start with the first two fixes—they resolve 90% of this specific error!

内容的提问来源于stack exchange，提问作者Stijn Meijers