pip安装含C++ Makefile的Python包时.so未生成导入报错排查

问题背景 #

开发了集成C代码的Python包，C代码通过Makefile完成编译，包内Python逻辑需要导入Makefile编译生成的.so共享库文件。目前已实现执行pip install myModule时自动触发Makefile运行，但安装完成后site-packages对应的包目录下找不到生成的.so文件，安装及导入阶段先后触发语法错误、导入错误。

项目目录结构 #

/myModule
├── /src
│   └── /myModule
│       ├── __init__.py
│       ├── main.py
│       └── /cppCode
│           ├── __init__.py
│           ├── wrapper.py
│           ├── C.cpp
│           ├── Makefile-conda
│           └── Makefile-gpu
├── LICENSE
├── requirements.txt
├── README.md
└── setup.py

编译生成的.so文件名为_C.so，在wrapper.py中完成导入。

初始报错信息 #

最初执行pip安装时抛出语法错误：

Installing collected packages: myModule
  *** Error compiling '/Users/user/anaconda3/lib/python3.8/site-packages/myModule/cppCode/wrapper.py'...
    File "/Users/user/anaconda3/lib/python3.8/site-packages/myModule/cppCode/wrapper.py", line 1
      import ._C as c
             ^
  SyntaxError: invalid syntax

当前setup.py配置 #

import os
import subprocess
from setuptools import setup, find_packages
from setuptools.command.build_ext import build_ext

CONDA_PREFIX = os.environ.get("CONDA_PREFIX", None)
WORKING_DIR = os.getcwd()


class Build(build_ext):
    """Customized setuptools build command - builds protos on build."""
    def run(self):
        run_makefile()
        os.chdir(WORKING_DIR)
        build_ext.run(self)


def makefile_command(gpu: bool):
    if not gpu:
        return ["conda run -n boost make -f Makefile-conda"]
    return ["conda run -n boost make -f Makefile-conda", "conda run -n boost make -f Makefile-gpu"]


def run_makefile():
    # 检测到conda环境时执行对应makefile编译
    if CONDA_PREFIX is not None:
        print("Download version with gpu.")

        # 检测GPU可用性
        import torch
        if not torch.cuda.is_available():
            print("Does not support GPU.\nBuild cpu.")
            gpu_available = False
        else:
            print("Support GPU.\nBuild GPU.")
            gpu_available = True

        # 执行Makefile编译
        os.chdir("src/myModule/cppCode")
        os.system("conda env create -f env.yml --force")
        cmd = '. ' + CONDA_PREFIX + '/etc/profile.d/conda.sh && conda activate boost'
        subprocess.call(cmd, shell=True, executable='/bin/bash')
        for command in makefile_command(gpu_available):
            process = subprocess.Popen(
                command.split(), stdout=subprocess.PIPE)
            output = process.stdout.read()
            print(output)
            output, error = process.communicate()
            print(output)
            print(error)

    else:
        print("Does not use Conda environment or Linux.\nDownload version without gpu")


if __name__ == '__main__':
    # 读取配置文件
    with open("requirements.txt") as f:
        requirements = [l.strip() for l in f.readlines()]

    with open("README.md") as r:
        readme = r.read()

    setup(
        name="myModule",
        version="0.0.1",
        license="GPL",
        author="Author",
        description_file="README.md",
        license_files="LICENSE.rst",
        install_requires=requirements,
        packages=find_packages('src'),
        python_requires=">=3.6.8",
        include_package_data=True,
        has_ext_modules=lambda: True,
        package_dir={"": "src"},
        cmdclass={
            'build_ext': Build
        },
        easy_install="ok_zip"
    )

环境说明 #

当前包仅支持Linux环境，因依赖boost-python，编译环境通过Conda管理，支持pip安装。曾尝试使用setuptools.extensions.Extension实现编译逻辑，但对该用法不熟悉，且/cppCode目录下包含Python文件，不确定该方案是否适用，判断问题根源在setup.py的配置逻辑。

运行环境 #

• python 3.8
• Linux
• anaconda3
• g++

问题更新 #

此前的SyntaxError是相对导入写法错误导致，已将wrapper.py中的导入语句修改为：

from . import _C as c

修改后执行pip安装不再抛出错误，但编写测试脚本调用wrapper.py时触发如下ImportError：

Traceback (most recent call last):
  File "/Users/user/test.py", line 1, in <module>
    import myModule.cppCode.wrapper as wrap
  File "/Users/user/anaconda3/lib/python3.8/site-packages/myModule/cppCode/wrapper.py", line 1, in <module>
    from . import _C as c
ImportError: cannot import name '_C' from 'myModule.cppCode' (/Users/user/anaconda3/lib/python3.8/site-packages/myModule/cppCode/wrapper.py__init__.py)

尽管pip安装过程无报错，但确认安装完成后.so文件仍未生成/拷贝到对应安装目录。

内容的提问来源于stack exchange，提问作者kabam