Mac M1环境下PySpark Shell与spark-submit的Numpy架构兼容问题求助

问题背景 #

通过pip3安装PySpark、great_expectations等依赖后，使用spark-submit运行测试代码时出现架构不兼容错误：Numpy的arm64版本无法在x86_64环境下加载。Python解释器为通用二进制（支持arm64e和x86_64），但PySpark交互式Shell可正常导入Numpy；安装x86_64版本的Numpy后spark-submit恢复正常，但PySpark Shell出现异常，需找到两者均能正常使用Numpy的解决方案。

测试代码 #

from pyspark.sql import SparkSession
import numpy as np
import pandas as pd


logFile = "/Users/.../spark/spark-3.4.0-bin-hadoop3/README.md"  # 替换为本地有效文件路径
spark = SparkSession.builder.appName("SimpleApp").getOrCreate()
logData = spark.read.text(logFile).cache()

numAs = logData.filter(logData.value.contains('a')).count()
numBs = logData.filter(logData.value.contains('b')).count()

print("Lines with a: %i, lines with b: %i" % (numAs, numBs))

spark.stop()

spark-submit运行报错信息 #

Traceback (most recent call last):
  File "/Users/.../Library/Python/3.9/lib/python/site-packages/numpy/core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/Users/.../Library/Python/3.9/lib/python/site-packages/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/Users/.../Library/Python/3.9/lib/python/site-packages/numpy/core/overrides.py", line 8, in <module>
    from numpy.core._multiarray_umath import (
ImportError: dlopen(/Users/.../Library/Python/3.9/lib/python/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so, 0x0002): tried: '/Users/.../Library/Python/3.9/lib/python/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/.../Library/Python/3.9/lib/python/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so' (no such file), '/Users/.../Library/Python/3.9/lib/python/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/.../Developer/Repos/night-eagle/platform/python/apps/purposing/sample.py", line 2, in <module>
    import numpy as np
  File "/Users/.../Library/Python/3.9/lib/python/site-packages/numpy/__init__.py", line 139, in <module>
    from . import core
  File "/Users/.../Library/Python/3.9/lib/python/site-packages/numpy/core/__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.9 from "/Library/Developer/CommandLineTools/usr/bin/python3"
  * The NumPy version is: "1.25.0"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: dlopen(/Users/.../Library/Python/3.9/lib/python/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so, 0x0002): tried: '/Users/.../Library/Python/3.9/lib/python/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/.../Library/Python/3.9/lib/python/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so' (no such file), '/Users/.../Library/Python/3.9/lib/python/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

Python架构检查结果 #

...@ATL100000847245 bin % file $(which python3)
/usr/bin/python3: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
/usr/bin/python3 (for architecture x86_64): Mach-O 64-bit executable x86_64
/usr/bin/python3 (for architecture arm64e): Mach-O 64-bit executable arm64e

PySpark Shell正常运行示例 #

Python 3.9.6 (default, Mar 10 2023, 20:16:38) 
[Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/06/21 18:04:36 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/_ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.4.0
      /_/

Using Python version 3.9.6 (default, Mar 10 2023 20:16:38)
Spark context Web UI available at http://a4efxml61027.eis.equifax.com:4040
Spark context available as 'sc' (master = local[*], app id = local-1687385077853).
SparkSession available as 'spark'.
>>> import numpy as np
>>> a = np.array([12,23,23])
>>> print(a)
[12 23 23]

解决方案 #

方案1：统一PySpark运行时架构 #

强制spark-submit和PySpark Shell使用同一架构运行，同时安装对应架构的Numpy：

选择arm64架构

1. 卸载现有Numpy：

   pip uninstall numpy -y
   

2. 用arm64架构安装Numpy：

   arch -arm64 pip install numpy
   

3. 运行命令时指定架构：

   # 执行spark-submit
   arch -arm64 spark-submit your_script.py
   # 启动PySpark Shell
   arch -arm64 pyspark
   

选择x86_64架构（依赖Rosetta2）

1. 卸载现有Numpy：

   pip uninstall numpy -y
   

2. 用x86_64架构安装Numpy：

   arch -x86_64 pip install numpy
   

3. 运行命令时指定架构：

   # 执行spark-submit
   arch -x86_64 spark-submit your_script.py
   # 启动PySpark Shell
   arch -x86_64 pyspark
   

方案2：创建独立虚拟环境 #

通过虚拟环境隔离依赖，确保同一环境下的PySpark Shell和spark-submit使用相同的依赖版本与架构：

1. 创建虚拟环境：

   python3 -m venv spark_env
   

2. 激活虚拟环境：

   source spark_env/bin/activate
   

3. 重新安装所有依赖：

   pip install pyspark great_expectations numpy
   

4. 在虚拟环境中运行命令：

   spark-submit your_script.py
   pyspark
   

内容的提问来源于stack exchange，提问作者Vishwanath560