Mac M2平台Cython编译后二进制不兼容问题的成因与解决

错误信息 #

ValueError: libs.portfolio_manager.lib.adjustment.Adjustment size changed, may indicate binary incompatibility. Expected 56 from C header, got 24 from PyObject

模块目录结构 #

/src
  /libs
    /portfolio_manager
      /lib/
        __init__.py
        adjustment.pyx
        adjustment.pxd
setup.py

setup.py核心代码片段 #

import os
import versioneer
from setuptools import Extension, setup


class LazyBuildExtCommandClass(dict):
    """
    Lazy command class that defers operations requiring Cython and numpy until
    they've actually been downloaded and installed by setup_requires.
    """

    def __contains__(self, key):
        return key == "build_ext" or super(LazyBuildExtCommandClass, self).__contains__(
            key
        )

    def __setitem__(self, key, value):
        if key == "build_ext":
            raise AssertionError("build_ext overridden!")
        super(LazyBuildExtCommandClass, self).__setitem__(key, value)

    def __getitem__(self, key):
        if key != "build_ext":
            return super(LazyBuildExtCommandClass, self).__getitem__(key)

        import numpy
        from Cython.Distutils import build_ext as cython_build_ext

        # Cython_build_ext isn't a new-style class in Py2.
        class build_ext(cython_build_ext, object):
            """
            Custom build_ext command that lazily adds numpy's include_dir to
            extensions.
            """

            def build_extensions(self):
                """
                Lazily append numpy's include directory to Extension includes.
                This is done here rather than at module scope because setup.py
                may be run before numpy has been installed, in which case
                importing numpy and calling `numpy.get_include()` will fail.
                """
                numpy_incl = numpy.get_include()
                for ext in self.extensions:
                    ext.include_dirs.append(numpy_incl)

                super(build_ext, self).build_extensions()

        return build_ext


ext_modules = [
    Extension(
        name="src.libs.portfolio_manager.lib.adjustment",
        sources=["src/libs/portfolio_manager/lib/adjustment.pyx"],
        define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")],
    ),
]


setup(
    ext_modules=ext_modules,
    cmdclass=LazyBuildExtCommandClass(versioneer.get_cmdclass()),
    package_data={
        root.replace(os.sep, "."): ["*.pyi", "*.pyx", "*.pxi", "*.pxd"]
        for root, dirnames, filenames in os.walk("src/libs/portfolio_manager")
        if "__pycache__" not in root
    },
)

成因分析 #

这个错误的核心是Cython生成的二进制模块中，C结构体的大小与Python运行时预期的大小不匹配，常见触发场景：

1. Numpy版本不匹配：编译模块时使用的Numpy版本，和运行时环境的Numpy版本不一致。Numpy的内部API结构体大小可能随版本更新变化，即使设置了NPY_NO_DEPRECATED_API，如果版本不对应也会导致结构大小计算偏差。
2. 架构不兼容：Mac M2是arm64架构，若编译时使用了x86_64环境（比如通过Rosetta启动Conda），或者Conda环境的架构与系统不匹配，会导致编译器生成的结构体大小和运行时解析的不一致。
3. 残留编译产物：之前编译的旧.so、.c文件未清理，新编译的代码与旧缓存文件冲突。
4. Cython定义不一致：adjustment.pxd中定义的C结构体（如Adjustment）与adjustment.pyx中的实现、或外部C头文件的定义不匹配，字段数量/类型存在差异。

解决步骤 #

• 统一Numpy版本并重新编译

  1. 激活目标Conda虚拟环境，卸载现有Numpy：conda remove numpy
  2. 安装指定版本的Numpy（推荐用当前稳定版）：conda install numpy
  3. 清理所有编译残留：删除模块目录下的.so、.c文件，以及build目录、__pycache__文件夹
  4. 重新执行编译命令：python setup.py build_ext --inplace

• 验证环境架构匹配

  1. 检查当前Conda环境的架构：运行conda info，查看platform字段是否为osx-arm64
  2. 若为x86_64环境，重新创建arm64架构的Conda环境：conda create -n your_env python=3.x -c conda-forge（conda-forge默认提供arm64包）
  3. 激活新环境后重新安装依赖并编译

• 修正Numpy宏定义
  检查setup.py中的define_macros参数，确保NPY_NO_DEPRECATED_API对应的版本与当前安装的Numpy匹配。例如使用Numpy 1.22+时，可以改为("NPY_NO_DEPRECATED_API", "NPY_LATEST_STABLE_API")，或指定对应版本的宏（如NPY_1_22_API_VERSION）

• 检查Cython定义一致性
  对比adjustment.pxd和adjustment.pyx中Adjustment结构体的定义，确保字段数量、类型完全一致；如果涉及外部C头文件，确认头文件中的结构体定义与pxd文件完全匹配

内容的提问来源于stack exchange，提问作者Jason Strimpel