AWS EMR提交Spark作业时陷入UNDEFINED状态求助

问题描述 #

在AWS EMR（版本emr-5.26.0）的m4.xlarge实例上执行以下spark-submit脚本提交作业：

#!/bin/sh

# Define variables to get script parameters
MAIN_SPARK_URI=$1 # looks like s3://bucket/app/src/main.py
MODELS_BASE_URI=$2 # looks like s3://bucket/app/models/models.pkl
APP_EGG_URI=$3 # looks like s3://bucket/app/src/app.egg
CONFIG_FILE_URI=$4 # looks like s3://bucket/app/src/config.ini
INPUT_DATA_URI=$5 # looks like s3://data-bucket/app/raw/consumption
VIRTUAL_ENVIRONMENT=$6 # looks like s3://bucket/app/virtualenv

# Launch a spark submit
spark-submit \\
--master yarn \\
--deploy-mode cluster \\
--name water_consumption_job \\
--conf spark.driver.cores=1 \\
--conf spark.driver.memoryOverhead=6G \\
--conf spark.driver.memory=6G \\
--conf spark.spark.executor.instances=4 \\
--conf spark.executor.cores=4 \\
--conf spark.executor.memoryOverhead=3G \\
--conf spark.executor.memory=3G \\
--conf spark.shuffle.service.enabled=false \\
--conf spark.dynamicAllocation.enabled=false \\
--conf spark.yarn.submit.waitAppCompletion=false \\
--conf spark.sql.caseSensitive=true \\
--conf spark.yarn.appMasterEnv.PYTHON_EGG_CACHE=. \\
--conf spark.yarn.appMasterEnv.CONFIG_FILE_LOCATION=${CONFIG_FILE_URI} \\
--conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=/usr/bin/python3 \\
--conf spark.executorEnv.PYTHON_EGG_CACHE=. \\
--conf spark.pyspark.virtualenv.bin.path=./venv/bin/python \\
--conf spark.pyspark.python=./venv/bin/python \\
--conf spark.pyspark.driver.python=./venv/bin/python \\
--archives $VIRTUAL_ENVIRONMENT/venv.tar.gz#venv \\
--files ${CONFIG_FILE_URI} \\
--py-files ${APP_EGG_URI} \\
${MAIN_SPARK_URI} \\
--pipeline water_consumptions \\
--input $INPUT_DATA_URI \\
--models $MODELS_BASE_URI/models.pkl
--collection mongo://consumptions/waterConsumptions \\

作业提交后命令直接退出并返回0，但实际未执行任何逻辑，日志显示作业处于UNDEFINED状态：

22/10/25 17:43:46 INFO Client: 
     client token: N/A
     diagnostics: [Tue Oct 25 17:43:46 +0000 2022] Application is Activated, waiting for resources to be assigned for AM.  Details : AM Partition = CORE ; Partition Resource = <memory:24576, vCores:16> ; Queue's Absolute capacity = 100.0 % ; Queue's Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; 
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1666719826565
     final status: UNDEFINED
     tracking URL: http://<something>/proxy/application_1666719665916_0001/
     user: hadoop
22/10/25 17:43:46 INFO ShutdownHookManager: Shutdown hook called
22/10/25 17:43:46 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-d83396ad-7b53-4e8b-a2e8-e9f8374f36a2
22/10/25 17:43:46 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-6a969106-81af-465a-a1eb-2eabd628f4b1
Command exiting with ret '0'

指定的Python主脚本未执行，无ACCEPTED/FINISHED状态日志输出。

排查与解决步骤 #

1. 修复脚本续行符错误 #

脚本中--models $MODELS_BASE_URI/models.pkl行末尾缺少续行符\\，导致后续的--collection参数未被spark-submit识别，作业参数解析失败，直接导致启动异常。修正后该行应为：

--models $MODELS_BASE_URI/models.pkl \\

2. 调整资源配置适配实例规格 #

m4.xlarge实例为4 vCPU、16GB内存，当前配置存在资源超额申请：

• Executor配置spark.executor.instances=4 + spark.executor.cores=4，单实例vCPU仅4核，无法同时承载4个各占4核的Executor，YARN无法分配资源，ApplicationMaster无法启动
• 驱动内存配置spark.driver.memory=6G + spark.driver.memoryOverhead=6G，总占用12G，虽能容纳但挤压了Executor可用资源

建议调整为以下配置：

--conf spark.driver.cores=1 \\
--conf spark.driver.memory=4G \\
--conf spark.driver.memoryOverhead=2G \\
--conf spark.executor.instances=2 \\
--conf spark.executor.cores=2 \\
--conf spark.executor.memory=4G \\
--conf spark.executor.memoryOverhead=2G \\

3. 验证虚拟环境归档有效性 #

确保s3://bucket/app/virtualenv/venv.tar.gz是完整的Python 3虚拟环境归档，包含所有依赖包。可在EMR节点上执行以下命令验证：

aws s3 cp $VIRTUAL_ENVIRONMENT/venv.tar.gz . && tar -tvf venv.tar.gz

EMR 5.26.0默认Python版本为2.7，需保证虚拟环境基于Python 3构建，与配置中PYSPARK_PYTHON=/usr/bin/python3匹配。

4. 修改作业等待配置 #

当前spark.yarn.submit.waitAppCompletion=false会让spark-submit提交后立即退出，无法获取作业后续日志。改为true可让命令阻塞至作业结束，便于排查：

--conf spark.yarn.submit.waitAppCompletion=true \\

5. 查看YARN详细日志 #

通过日志中的tracking URL访问YARN ResourceManager页面，查看ApplicationMaster的启动日志；或登录EMR集群节点，查看/var/log/yarn/目录下的相关日志文件，获取资源分配失败、虚拟环境加载错误等详细信息。

内容的提问来源于stack exchange，提问作者Mistapopo