在Dataproc+PySpark中使用spaCy遇numpy二进制兼容错误求助

问题背景 #

此前因本地磁盘空间不足，创建了全新Dataproc集群用于spaCy命名实体识别，集群创建命令如下：

gcloud dataproc clusters create spacy_tests \
--autoscaling-policy policy-dbeb \
--enable-component-gateway \
--region us-central1 \
--zone us-central1-c \
--master-machine-type n1-standard-4 \
--master-boot-disk-size 500 \
--num-workers 2 \
--worker-machine-type n1-standard-4 \
--worker-boot-disk-size 500 \
--num-secondary-workers 2 \
--secondary-worker-boot-disk-size 500 \
--num-secondary-worker-local-ssds 0 \
--image-version 1.5-debian10 \
--properties dataproc:pip.packages=spacy==3.2.1,numpy==1.19.5,dataproc:efm.spark.shuffle=primary-worker \
--optional-components ANACONDA,JUPYTER,DOCKER \
--project mentor-pilot-project

错误详情 #

运行代码时触发以下ValueError：

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-9a88da6b7731> in <module>
     39 
     40 # loading spaCy model and broadcasting it
---> 41 broadcasted_nlp = spark.sparkContext.broadcast(load_spacy_model())
     42 
     43 print("DATA READING (OR MANUAL DATA GENERATION)...")

<ipython-input-1-9a88da6b7731> in load_spacy_model()
     20 
     21 def load_spacy_model():
---> 22     import spacy
     23     print("\tLoading spacy model...")
     24     return spacy.load("./spacy_model")  # This model exists locally

/opt/conda/anaconda/lib/python3.7/site-packages/spacy/__init__.py in <module>
      9 
     10 # These are imported as part of the API
---> 11 from thinc.api import prefer_gpu, require_gpu, require_cpu  # noqa: F401
     12 from thinc.api import Config
     13 

/opt/conda/anaconda/lib/python3.7/site-packages/thinc/api.py in <module>
      1 from .config import Config, registry, ConfigValidationError
----> 2 from .initializers import normal_init, uniform_init, glorot_uniform_init, zero_init
      3 from .initializers import configure_normal_init
      4 from .loss import CategoricalCrossentropy, L2Distance, CosineDistance
      5 from .loss import SequenceCategoricalCrossentropy

/opt/conda/anaconda/lib/python3.7/site-packages/thinc/initializers.py in <module>
      2 import numpy
      3 
----> 4 from .backends import Ops
      5 from .config import registry
      6 from .types import FloatsXd, Shape

/opt/conda/anaconda/lib/python3.7/site-packages/thinc/backends/__init__.py in <module>
      6 
      7 from .ops import Ops
----> 8 from .cupy_ops import CupyOps, has_cupy
      9 from .numpy_ops import NumpyOps
     10 from ._cupy_allocators import cupy_tensorflow_allocator, cupy_pytorch_allocator

/opt/conda/anaconda/lib/python3.7/site-packages/thinc/backends/cupy_ops.py in <module>
     17 from .. import registry
     18 from .ops import Ops
---> 19 from .numpy_ops import NumpyOps
     20 from . import _custom_kernels
     21 from ..types import DeviceTypes

/opt/conda/anaconda/lib/python3.7/site-packages/thinc/backends/numpy_ops.pyx in init thinc.backends.numpy_ops()

ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

原因分析 #

该错误属于典型的Python扩展库二进制兼容性问题：

• spaCy依赖的thinc库通过Cython编译，编译时绑定了特定版本numpy的C结构体定义（numpy.ndarray的内存布局）。
• 集群创建时同时启用了ANACONDA组件，且通过pip.packages指定了numpy==1.19.5，导致conda自带的numpy版本与pip安装的numpy版本冲突，或者pip安装的numpy与thinc编译时依赖的numpy版本不匹配，最终引发结构体大小不兼容的错误。

解决方案 #

方案1：统一依赖安装渠道，使用conda管理包 #

由于集群启用了ANACONDA组件，优先用conda而非pip安装依赖，避免混合渠道导致的版本冲突。修改集群创建命令的properties部分：

--properties dataproc:conda.packages=spacy=3.2.1,numpy=1.19.5,dataproc:efm.spark.shuffle=primary-worker

注：conda包版本格式用=而非==，需确保指定的版本在conda仓库中存在。

方案2：移除手动指定的numpy版本，适配Anaconda默认版本 #

删除pip.packages中的numpy==1.19.5，让Anaconda自带的numpy版本与spaCy自动兼容：

--properties dataproc:pip.packages=spacy==3.2.1,dataproc:efm.spark.shuffle=primary-worker

若spaCy与默认numpy版本不兼容，可查询Dataproc 1.5-debian10的Anaconda默认numpy版本，再安装对应兼容的spaCy版本。

方案3：强制重新安装兼容的numpy版本 #

在集群初始化脚本中添加步骤，卸载冲突的numpy并重新安装指定版本：

1. 创建初始化脚本fix_numpy.sh：

#!/bin/bash
pip uninstall -y numpy
pip install numpy==1.19.5 --force-reinstall

2. 修改集群创建命令，添加初始化脚本：

gcloud dataproc clusters create spacy_tests \
--autoscaling-policy policy-dbeb \
--enable-component-gateway \
--region us-central1 \
--zone us-central1-c \
--master-machine-type n1-standard-4 \
--master-boot-disk-size 500 \
--num-workers 2 \
--worker-machine-type n1-standard-4 \
--worker-boot-disk-size 500 \
--num-secondary-workers 2 \
--secondary-worker-boot-disk-size 500 \
--num-secondary-worker-local-ssds 0 \
--image-version 1.5-debian10 \
--properties dataproc:pip.packages=spacy==3.2.1,numpy==1.19.5,dataproc:efm.spark.shuffle=primary-worker \
--optional-components ANACONDA,JUPYTER,DOCKER \
--project mentor-pilot-project \
--initialization-actions gs://your-bucket/fix_numpy.sh

需将gs://your-bucket/fix_numpy.sh替换为实际的GCS存储路径。

内容的提问来源于stack exchange，提问作者David Espinosa