Django可执行文件调用Spark Job报错：路径格式异常排查

执行的spark-submit命令 #

/opt/spark-3.5.5-bin-hadoop3/bin/spark-submit --master local --deploy-mode client --conf "spark.ui.enabled=false" --conf "spark.ui.showConsoleProgress=false" --conf "spark.dynamicAllocation.enabled=false" --conf "spark.rdd.compress=false" --conf "spark.driver.memory=4g" --conf "spark.executor.memory=8g" --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" /Users/user1/Project/process-engine/route.py > /app/server/EAP/rasLite/engine/raslight/spark_submit_log/20250424/772_1.0_174702842893.log 2>&1 "{'processNo': '772', 'versionNo': '1.0', 'jsonData': '', 'executionDate': '', 'skipError': 'N', 'generated_executionid': '149897', 'isExecutionIdGenerated': 'True', 'executionId': '149897', 'isPreProcess': 'False'}" &

错误日志 #

Unknown command: 'C:/Users/user1/Project/process-engine/route.py'
Type 'ras.exe help' for usage.

相关背景 #

• Django项目打包为可执行文件后，通过API端点触发该spark-submit命令
• route.py的Unix风格路径是正确的，但错误日志显示为Windows风格路径
• 使用python manage.py runserver 8000正常运行时无异常，仅打包为可执行文件后出现问题
• 已设置SPARK_HOME环境变量，但问题未解决

执行命令的代码片段 #

command = f'/Users/user1/Project/process-engine/spark-3.5.5-bin-hadoop3/bin/spark-submit --master local --deploy-mode client --conf "spark.ui.enabled=false" --conf "spark.ui.showConsoleProgress=false" --conf "spark.dynamicAllocation.enabled=false" --conf "spark.rdd.compress=false" --conf "spark.driver.memory=4g" --conf "spark.executor.memory=8g" --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" /Users/user1/Project/process-engine/route.py > {finalFilePath} 2>&1'

final_command = command + " " + '\\"' + raw_body_decoded1 + '\\"' + " " + "&"

# Set the environment variables for the subprocess
env = os.environ.copy()
env['DJANGO_SETTINGS_MODULE'] = 'rasLightEngine.settings'
env['SPARK_HOME'] = 'C:/Users/user1/Project/process-engine/spark-3.5.5-bin-hadoop3'

subprocess.Popen(f'{final_command}', shell=True, env=env)

解决方案 #

1. 禁用打包工具的路径自动转换 #

打包工具（如PyInstaller）在Windows环境下打包时，会自动把Unix风格路径转为Windows格式，导致spark-submit无法识别。解决方式：

• 定义路径时使用原始字符串，避免自动转换：

  spark_submit_path = r'/Users/user1/Project/process-engine/spark-3.5.5-bin-hadoop3/bin/spark-submit'
  route_py_path = r'/Users/user1/Project/process-engine/route.py'
  

• 若用PyInstaller打包，添加参数禁用路径转换：

  pyinstaller --noconvert-paths your_script.spec
  

2. 改用subprocess参数列表调用，避免shell解析 #

shell=True依赖系统shell解析命令，容易出现路径转义问题。改用参数列表形式直接传递给spark-submit：

import subprocess
import os

# 拆分命令为参数列表，避免shell解析
spark_submit_cmd = [
    '/Users/user1/Project/process-engine/spark-3.5.5-bin-hadoop3/bin/spark-submit',
    '--master', 'local',
    '--deploy-mode', 'client',
    '--conf', 'spark.ui.enabled=false',
    '--conf', 'spark.ui.showConsoleProgress=false',
    '--conf', 'spark.dynamicAllocation.enabled=false',
    '--conf', 'spark.rdd.compress=false',
    '--conf', 'spark.driver.memory=4g',
    '--conf', 'spark.executor.memory=8g',
    '--conf', 'spark.serializer=org.apache.spark.serializer.KryoSerializer',
    '/Users/user1/Project/process-engine/route.py',
    raw_body_decoded1
]

# 用subprocess参数处理日志重定向，替代shell重定向
with open(finalFilePath, 'w') as log_file:
    subprocess.Popen(
        spark_submit_cmd,
        stdout=log_file,
        stderr=subprocess.STDOUT,
        env=env,
        shell=False
    )

3. 统一路径风格与SPARK_HOME设置 #

确保SPARK_HOME的路径风格和其他路径保持一致：

• 若运行环境是Unix/Linux，SPARK_HOME使用Unix风格路径：

  env['SPARK_HOME'] = '/Users/user1/Project/process-engine/spark-3.5.5-bin-hadoop3'
  

• 避免混合Windows和Unix风格路径，防止环境变量解析错误。

4. 验证打包后的文件路径映射 #

打包工具可能会把本地文件映射到临时目录，需确认route.py可被正确访问：

• 若route.py是项目内文件，打包时确保被包含（PyInstaller为例，在spec文件中添加datas=[('route.py', '.')]）
• 动态获取打包后的临时目录路径：

  import sys
  if getattr(sys, 'frozen', False):
      # 打包后的环境
      route_py_path = os.path.join(sys._MEIPASS, 'route.py')
  else:
      # 开发环境
      route_py_path = '/Users/user1/Project/process-engine/route.py'
  

内容的提问来源于stack exchange，提问作者Rudra Patel