AWS EMR 5.27集群Spark库更新及自定义JAR部署节点咨询

Great question—this comes up a lot when working with custom Spark extensions on EMR. Let’s break this down clearly based on how Spark runs in an EMR cluster:

First, Understand Driver vs. Executor Roles #

In EMR:

• The master node runs the Spark Driver, which coordinates your job, parses SQL, and manages overall execution flow.
• Core and task nodes run Spark Executors, which handle the heavy lifting: executing SQL query logic, running custom UDFs, reading/writing data via custom sources, etc.

Do You Need to Update All Nodes? #

It depends on what your custom Spark SQL JAR does, but in almost all cases, you’ll need to update every node (master, core, task). Here’s why:

• If your JAR includes custom UDFs, data sources, Spark SQL optimizer rules, or any logic tied to data processing, that code has to run on Executors (core/task nodes). If only the master has the updated JAR, Executors will still use the old version (or missing classes entirely), leading to ClassNotFoundException or inconsistent job behavior.
• Even if part of your logic runs on the Driver, if it references classes that Executors need (which is nearly always true for Spark SQL extensions), Executors must have the same JAR version to avoid compatibility mismatches.

Exceptions (Extremely Rare!) #

The only scenario where updating just the master might work is if your JAR is exclusively for Driver-side logic that never interacts with Executors—for example, a utility script that only generates Spark SQL queries but doesn’t define reusable UDFs or extensions. This is almost never the case for Spark SQL-focused JARs.

Recommended Approaches #

Instead of manually updating JARs across all nodes (which is error-prone), consider these better options:

1. Submit jobs with the --jars parameter
   When launching your Spark job, use spark-submit --jars s3://your-bucket/path/to/custom.jar ... (or a local path if the JAR is already synced to all nodes). Spark will automatically distribute this JAR to all Executors, ensuring every node uses the same version. This is the most flexible approach and avoids modifying the cluster’s default Spark installation.

2. Use EMR Cluster Steps to update all nodes
   If you need all jobs on the cluster to use the custom JAR, create a shell script that copies the JAR to /usr/lib/spark/jars/ on every node, then run it as an EMR Cluster Step. For example:

   # Script to copy custom JAR to all nodes
   sudo aws s3 cp s3://your-bucket/custom-spark-sql.jar /usr/lib/spark/jars/
   sudo chown root:root /usr/lib/spark/jars/custom-spark-sql.jar
   

   EMR will execute this script on every node in the cluster when you run it as a step.

Final Takeaway #

Don’t skip updating core/task nodes unless you’re 100% certain your JAR has zero Executor-side logic. The safest and most maintainable approach is to use --jars when submitting jobs, as it keeps your custom code isolated from the cluster’s base Spark setup.

内容的提问来源于stack exchange，提问作者Bostonian