咨询SPARK_SUBMIT_OPTIONS中<CPS>的含义（AWS EMR场景）

Great question! Let's break this down clearly:

First: What is ? #

<CPS> is a placeholder that stands for Colon Path Separator (or more broadly, system path separator). It’s designed to be replaced at runtime with the correct path separator character for your operating system:

• On Unix/Linux systems (like AWS EMR, which uses Linux under the hood), it gets replaced with a colon :
• On Windows systems, it would be replaced with a semicolon ;

This placeholder makes configuration files cross-platform friendly—you don’t have to hardcode path separators that only work on one OS.

Applying This to Your Zeppelin Config #

Looking at your specific snippet from /etc/zeppelin/conf/zeppelin-env.sh:

export SPARK_SUBMIT_OPTIONS="$SPARK_SUBMIT_OPTIONS \ --conf 'spark.executorEnv.PYTHONPATH=/usr/lib/spark/python/lib/py4j-src.zip:/usr/lib/spark/python/:<CPS>{{PWD}}/pyspark.zip<CPS>{{PWD}}/py4j-src.zip' \ --conf spark.yarn.isPython=true"

When this runs on AWS EMR, the <CPS> placeholders will be swapped out with :. The final spark.executorEnv.PYTHONPATH value that Spark uses will look like this:

/usr/lib/spark/python/lib/py4j-src.zip:/usr/lib/spark/python/:/{{PWD}}/pyspark.zip:/{{PWD}}/py4j-src.zip

The goal here is to:

• Include Spark’s default Python dependencies (like py4j) in the executor’s Python path
• Append custom pyspark.zip and py4j-src.zip files from Zeppelin’s current working directory, so Spark executors can access these custom resources when running Python jobs.

内容的提问来源于stack exchange，提问作者Dyno Fu