Github Actions构建PySpark包后模块导入失败求助

问题现象 #

通过Github Actions构建PySpark项目的zip包并推送到S3后，执行spark-submit --py-files src.zip main.py时触发ModuleNotFound错误，但本地使用Makefile打包后运行完全正常。

项目目录结构 #

── data_compaction
    ├── Makefile
    ├── main.py
    └── src
        ├── jobs
        │   ├── __init__.py
        │   ├── xyz.py
        │   └── abc.py
        └── utilities
            ├── __init__.py
            └── spark_foundation.py

Github Actions YAML配置 #

name: Build Artifacts
on:
  push:
    branches:
    - main
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.9"]
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: ${{ matrix.python-version }}
      - name: Make artifact directory
        run: mkdir -p ./dist
      - uses: actions/checkout@v2
      - name: Create Zip File
        uses: montudor/action-zip@v0.1.0
        with:
          args: sh -c "cd data_compaction && zip -r ../src.zip src/"
      - name: Push zip file to S3
        uses: qoqa/action-s3-cp@v1.1
        env:
          AWS_S3_BUCKET: 'dev-bucket'
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_REGION: 'us-east-1'
          AWS_S3_PATH: '/artifacts/src.zip'
          FILE: 'src.zip'
      - name: Push main file to S3
        uses: qoqa/action-s3-cp@v1.1
        env:
          AWS_S3_BUCKET: 'dev-bucket'
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_REGION: 'us-east-1'
          AWS_S3_PATH: '/artifacts/main.py'
          FILE: './data_compaction/main.py'

本地Makefile配置 #

build:
    rm -f -r ./dist
    mkdir ./dist
    cp main.py ./dist
    cd ./src && zip -r ../dist/src.zip .

main.py导入代码片段 #

import os
import sys

if os.path.exists('src.zip'):
    sys.path.insert(0, 'src.zip')
else:
    sys.path.insert(0, './src')
from utilities.spark_foundation import spark_session
from jobs.xyz import func1
from jobs.abc import func2

问题根源 #

本地打包命令是进入src目录后，将当前目录下的所有内容打包（zip -r ../dist/src.zip .），最终zip包的根目录直接包含utilities、jobs等模块目录；而Github Actions中的打包命令是在data_compaction目录下打包整个src目录（zip -r ../src.zip src/），导致zip包的根目录是src文件夹，模块实际在src/utilities、src/jobs路径下。当src.zip被加入sys.path后，Python无法直接找到utilities模块，因此触发ModuleNotFoundError。

解决方案 #

修改Github Actions中的打包步骤，和本地保持一致：进入src目录后再打包当前内容。调整后的Create Zip File步骤如下：

- name: Create Zip File
  uses: montudor/action-zip@v0.1.0
  with:
    args: sh -c "cd data_compaction/src && zip -r ../../src.zip ."

这样打包出来的zip包结构和本地完全一致，Python可以正确识别utilities、jobs模块。

内容的提问来源于stack exchange，提问作者Rohit Anil