如何通过Docker向spark-submit传递参数？问题排查与解决

The Problem #

I have a local Spark cluster running in Docker with 1 master and 3 workers. When I exec into the master container and run this spark-submit command directly, the WordCount example works perfectly and generates results in the output-5 directory:

bash-4.3# spark/bin/spark-submit --class com.oreilly.learningsparkexamples.mini.scala.WordCount --master spark://spark-master:7077 /opt/spark-apps/learning-spark-mini-example_2.11-0.0.1.jar /opt/spark-data/README.md /opt/spark-data/output-5

But when I try to submit the job from outside the container using this docker run command:

docker run --network docker-spark-cluster_spark-network -v /tmp/spark-apps:/opt/spark-apps --env SPARK_APPLICATION_JAR_LOCATION=$SPARK_APPLICATION_JAR_LOCATION --env SPARK_APPLICATION_MAIN_CLASS=$SPARK_APPLICATION_MAIN_CLASS -e APP_ARGS="/opt/spark-data/README.md /opt/spark-data/output-5" spark-submit:2.4.0

With these environment variables set correctly:

echo $SPARK_APPLICATION_JAR_LOCATION → /opt/spark-apps/learning-spark-mini-example_2.11-0.0.1.jar
echo $SPARK_APPLICATION_MAIN_CLASS → com.oreilly.learningsparkexamples.mini.scala.WordCount

I get this error:

Caused by: java.lang.ArrayIndexOutOfBoundsException: 0 at com.oreilly.learningsparkexamples.mini.scala.WordCount$.main(WordCount.scala:11)

This clearly means the WordCount app isn't receiving the input file path argument (index 0) that it needs.

The Fix #

After troubleshooting, the correct command to use is:

docker run --network docker-spark-cluster_spark-network -v /tmp/spark-apps:/opt/spark-apps --env SPARK_APPLICATION_JAR_LOCATION=$SPARK_APPLICATION_JAR_LOCATION --env SPARK_APPLICATION_MAIN_CLASS=$SPARK_APPLICATION_MAIN_CLASS --env SPARK_APPLICATION_ARGS="/opt/spark-data/README.md /opt/spark-data/output-6" spark-submit:2.4.0

Key Takeaway #

The issue wasn't with using -e (Docker's shorthand for --env)—that's still a valid way to set environment variables. The critical mistake was using APP_ARGS as the environment variable name instead of SPARK_APPLICATION_ARGS.

The spark-submit:2.4.0 image's internal scripts are purpose-built to look for SPARK_APPLICATION_ARGS to collect and pass arguments to your Spark application. Using a custom variable name like APP_ARGS meant the scripts never picked up the parameters, hence the ArrayIndexOutOfBoundsException when the app tried to access the missing input path.

内容的提问来源于stack exchange，提问作者aironman