无法通过自定义pip包调用已编译的bwa C库，本地安装正常

尝试将C库bwa打包为pip包bwaPython，安装过程显示成功，但调用包内方法时提示找不到bwa命令，报错：

FileNotFoundError: [Errno 2] No such file or directory: 'bwa'

本地直接安装bwa后可正常调用，但通过pip包调用时失效。

安装成功日志：

sultan@Sultans-MacBook-Pro bwaPython % python3 -m pip install .
Processing /Users/sultan/bwaPython
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: bwaPython
  Building wheel for bwaPython (setup.py) ... done
  Created wheel for bwaPython: filename=bwaPython-0.0.0-py3-none-any.whl size=1758 sha256=5eecae6b41154ec7c4463c578b526cc22473c0cd971aab8505f6526b37b1a350
  Stored in directory: /private/var/folders/q2/p3rkrrv9311c6pk2q4s7psxr0000gn/T/pip-ephem-wheel-cache-a7mlaws3/wheels/07/a6/db/ba14333ebca4820ddd16ca2c0be0a4b82fe15bbab3ab428607
Successfully built bwaPython
Installing collected packages: bwaPython
  Attempting uninstall: bwaPython
    Found existing installation: bwaPython 0.0.0
    Uninstalling bwaPython-0.0.0:
      Successfully uninstalled bwaPython-0.0.0
Successfully installed bwaPython-0.0.0

调用报错日志：

Python 3.10.6 (v3.10.6:9c7b4bd164, Aug  1 2022, 17:13:48) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bwaPython
>>> bwaPython.generate_bwa()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/bwaPython/subprocesses.py", line 5, in generate_bwa
    subprocess.call(["bwa", "index"])
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 345, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 969, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1845, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'bwa'

项目中subprocesses.py的调用代码：

def generate_bwa():
    subprocess.call(["bwa", "index"])

原setup.py尝试通过克隆仓库、编译并修改PATH来解决，但无效。

1. PATH修改仅对子shell生效：setup.py中通过subprocess执行的export PATH和source命令，只作用于当前子shell进程，不会影响用户的全局环境或后续启动的Python进程。
2. 编译后的bwa路径错误：setup.py中写死的~/bwaPython/bwa路径不成立——pip安装时会在临时目录处理编译，安装完成后临时目录会被清理，且该路径与实际安装后的bwa位置不符。
3. 未将bwa二进制文件放入可访问路径：编译后的bwa没有被安装到系统或Python环境的可执行路径中，导致Python的subprocess无法找到该命令。

方案1：将bwa安装到Python的bin目录（推荐） #

修改setup.py，将编译后的bwa二进制文件复制到Python的系统bin目录（该目录通常已在PATH中），无需手动修改环境变量：

import os
import shutil
import subprocess
import sys
from setuptools import setup, find_packages
from setuptools.command.install import install

def compile_and_install_bwa():
    # 使用临时目录编译bwa，避免残留文件
    import tempfile
    with tempfile.TemporaryDirectory() as tmpdir:
        os.chdir(tmpdir)
        # 克隆bwa仓库
        subprocess.check_call(["git", "clone", "https://github.com/lh3/bwa.git"])
        os.chdir("bwa")
        # 编译bwa
        subprocess.check_call(["make"])
        # 获取Python的bin目录路径
        target_bin = os.path.join(sys.prefix, "bin")
        os.makedirs(target_bin, exist_ok=True)
        # 复制编译好的bwa到目标目录
        shutil.copy("bwa", target_bin)

class CustomInstall(install):
    def run(self):
        compile_and_install_bwa()
        super().run()

setup(
    name="bwaPython",
    version="0.0.1",
    cmdclass={"install": CustomInstall},
    packages=find_packages(),
)

修改后，subprocesses.py无需改动，直接调用bwa即可：

import subprocess

def generate_bwa():
    subprocess.call(["bwa", "index"])

方案2：将bwa嵌入Python包内 #

如果不想依赖系统PATH，可以将bwa二进制文件放入Python包目录，调用时指定绝对路径：

修改setup.py #

import os
import shutil
import subprocess
import sys
from setuptools import setup, find_packages
from setuptools.command.install import install

def compile_and_install_bwa():
    import tempfile
    with tempfile.TemporaryDirectory() as tmpdir:
        os.chdir(tmpdir)
        subprocess.check_call(["git", "clone", "https://github.com/lh3/bwa.git"])
        os.chdir("bwa")
        subprocess.check_call(["make"])
        # 将bwa复制到包内的bin目录
        package_bin = os.path.join(os.path.dirname(os.path.abspath(__file__)), "bwaPython", "bin")
        os.makedirs(package_bin, exist_ok=True)
        shutil.copy("bwa", package_bin)

class CustomInstall(install):
    def run(self):
        compile_and_install_bwa()
        super().run()

setup(
    name="bwaPython",
    version="0.0.1",
    cmdclass={"install": CustomInstall},
    packages=find_packages(),
    # 声明包内的二进制文件，确保安装时被复制
    package_data={
        "bwaPython": ["bin/bwa"],
    },
)

修改subprocesses.py #

import os
import subprocess

def generate_bwa():
    # 获取包内bwa的绝对路径
    package_dir = os.path.dirname(os.path.abspath(__file__))
    bwa_path = os.path.join(package_dir, "bin", "bwa")
    # 确保bwa有执行权限
    os.chmod(bwa_path, 0o755)
    subprocess.call([bwa_path, "index"])

额外注意事项 #

• 移除原setup.py中所有修改PATH和执行source的代码，这些操作完全无效。
• 建议添加版本号到setup.py，方便后续更新维护。
• 如果需要支持多平台，需考虑不同系统的编译差异（如Windows需额外处理，但bwa主要用于类Unix系统）。

内容的提问来源于stack exchange，提问作者Sultan Arapov