pip安装Python包时，如何让编译生成的共享对象文件自动同步到安装目录

Great question! You’ve correctly identified the core issue: MANIFEST.in is parsed during the build phase (when pip creates a wheel/sdist package), but your shared library isn’t compiled until the install phase. This means the first install misses the file, while the second install picks it up because it already exists.

Let’s break down the best solutions to fix this, aligned with setuptools/pip best practices:

Root Cause Deep Dive #

When you run pip install ., pip follows this flow by default:

1. Build Isolation: Creates a temporary environment, copies files matching MANIFEST.in into it.
2. Build Phase: Runs setuptools’ build command to prepare artifacts.
3. Package Phase: Creates a wheel/sdist from the build artifacts.
4. Install Phase: Copies the wheel’s contents to your conda environment.

Your compilation runs in the install phase (via CustomInstall), which happens after the build/package phases. So the new shared library never makes it into the wheel, hence missing from the first install.

Solution 1: Compile During the Build Phase (Recommended) #

The cleanest fix is to move your compilation to the build phase (the proper stage for generating build artifacts). This ensures the shared library exists when MANIFEST.in is parsed, so it gets included in the wheel.

Modify your setup.py to add a custom build command:

import subprocess
from typing import List
from setuptools import find_namespace_packages, setup
from setuptools.command.install import install
from setuptools.command.build import build

class CustomBuild(build):
    def run(self):
        # First compile the shared library
        with open("build.log", "w") as f:
            # Use check=True to fail fast if compilation errors
            subprocess.run(["./compile.sh"], stdout=f, check=True)
        # Run the default build process (handles MANIFEST.in, packaging, etc.)
        super().run()

class CustomInstall(install):
    def run(self):
        # Keep this only if you need install-phase logging/operations
        with open("install.log", "w") as f:
            f.write("Installation completed successfully\n")
        super().run()

requirements: List[str] = [
    # Your dependencies here
]

setup(
    author="",
    python_requires=">=3.11",
    install_requires=requirements,
    name="mypkg",
    license="",
    packages=find_namespace_packages(include=["mypkg", "mypkg.*"]),
    cmdclass={
        "build": CustomBuild,
        "install": CustomInstall  # Remove if you don't need install-phase logic
    },
    include_package_data=True,
    zip_safe=False,
)

Why This Works #

• Follows setuptools’ intended lifecycle: Build artifacts are created in the build phase, then packaged into the wheel.
• Works with pip’s default build isolation mode—no extra flags needed.
• The shared library will be included in the wheel, making it distributable to other environments too.

Solution 2: Manually Copy the Library (Quick Fix) #

If you want to avoid changing the build flow, you can manually copy the compiled library to the install directory after compilation. This bypasses MANIFEST.in entirely.

Update your CustomInstall class:

import subprocess
import shutil
import glob
from typing import List
from setuptools import find_namespace_packages, setup
from setuptools.command.install import install

class CustomInstall(install):
    def run(self):
        # 1. Compile the shared library
        with open("install.log", "w") as f:
            subprocess.run(["./compile.sh"], stdout=f, check=True)
        # 2. Get the target install path (site-packages/mypkg)
        target_dir = f"{self.install_lib}/mypkg"
        # 3. Copy all libc.* files to the target
        for lib_file in glob.glob("libc.*"):
            print(f"Copying {lib_file} to {target_dir}")
            shutil.copy(lib_file, target_dir)
        # 4. Run the default install process
        super().run()

# Rest of your setup.py code remains the same

Notes #

• This directly copies the file to your conda environment’s site-packages/mypkg folder, so it’s available immediately after install.
• Downside: The shared library won’t be included in generated wheels (since wheels are built before compilation). Use this only if you don’t need to distribute the wheel.

Solution 3: Compile Directly to the Package Directory #

Adjust your compilation script to output the shared library directly into the mypkg folder, then use package_data to tell setuptools to include it.

1. Modify compile.sh: Change the output path to mypkg/. For example, if your compile command is:

   gcc -shared -o libc.so c_files/*.c
   

   Update it to:

   gcc -shared -o mypkg/libc.so c_files/*.c
   

2. Update setup.py: Add a package_data entry to explicitly include the library:

   setup(
       # ... other config ...
       include_package_data=True,
       package_data={"mypkg": ["libc.*"]},  # Explicitly declare the library files
       zip_safe=False,
   )
   

Why This Works #

• include_package_data=True tells setuptools to include files listed in package_data (or MANIFEST.in) when installing.
• Since the library is in the mypkg folder, it’s treated as part of the package and copied to the install directory automatically.

Testing the Fix #

Before testing, clean up any residual files from previous installs:

pip uninstall -y mypkg
rm -rf build/ dist/ mypkg.egg-info/ libc.*

Then run pip install . and check your conda environment’s site-packages/mypkg directory—your shared library should be there on the first try!

内容来源于stack exchange