如何在Setuptools可编辑安装中实现并行构建？

Great question — this is a super annoying gap in setuptools' functionality that's persisted for years, and I've run into the exact same issue with editable installs needing parallel builds for C extensions. Let's break down why your existing approach fails and walk through a few working solutions:

Why Your Current Method Breaks #

The error you're seeing (could not create _my_package_ext/...so) happens because when you run develop or pip install -e, setuptools tries to copy the built extension files from the build/lib... directory directly into your source tree (into _my_package_ext). If that target directory doesn't exist in your source code, the copy operation fails. The build -j4 command only builds the extensions in the build directory—it doesn't create the necessary folder in your repo root.

Working Solutions #

1. Pass Parallel Build Flags Directly via Pip (Best for Pip Users)

You can use pip's --global-option flag to pass the -j4 (or whatever number of cores you want) directly to the build_ext step that runs during the editable install. This lets you do everything in one command, no pre-build required:

pip install -e . --user --global-option="build_ext" --global-option="-j4"

Pip will pass these options to setuptools when it runs the develop command, triggering a parallel build of your extensions right in the editable install process.

2. Pre-Build with Parallelism + Create Target Directory First

If you still want to pre-build before running the editable install, just manually create the target directory that setuptools expects to copy into. For your error, that's _my_package_ext:

mkdir -p _my_package_ext
python setup.py build -j4
python setup.py develop

Or with pip:

mkdir -p _my_package_ext
python setup.py build -j4
pip install -e . --user

This fixes the "no such file or directory" error because the target folder now exists for setuptools to copy the built .so file into.

3. Modify setup.py to Auto-Use Parallel Builds

For a more permanent solution, you can tweak your setup.py to automatically use parallel builds by default, or read a parallelism value from an environment variable. Here's how to do it by overriding the build_ext command:

import os
import multiprocessing
from setuptools import setup
from setuptools.command.build_ext import build_ext

class ParallelBuildExt(build_ext):
    def build_extensions(self):
        # Use CPU count by default, or allow overriding via env var
        self.parallel = int(os.environ.get('PARALLEL_JOBS', multiprocessing.cpu_count()))
        super().build_extensions()

# Rest of your setup configuration
setup(
    name="My_Package",
    # ... your other setup args ...
    cmdclass={'build_ext': ParallelBuildExt}
)

Then, when you run the editable install, you can either let it use all your cores, or set the number explicitly:

# Use all cores
pip install -e . --user

# Or set a specific number
PARALLEL_JOBS=4 pip install -e . --user

This way you never have to remember to pass -j4 again—it's handled automatically.

内容的提问来源于stack exchange，提问作者marcman