CentOS 7环境下安装Scrapy时收集Twisted失败求助

Hey there! Let's work through this Twisted installation hang issue together. First, let's tackle your core question:

是不是Twisted的压缩格式导致的问题？ #

Probably not directly. The .tar.bz2 is just a standard source archive—on Linux, pip often falls back to compiling packages from source if pre-built binary packages (like .whl files) aren't available. The hang you're seeing is far more likely caused by missing compilation dependencies that Twisted needs to build successfully, or an outdated pip that isn't prioritizing binary packages.

On Windows, you didn't run into this because pip automatically downloads pre-compiled .whl files for most packages, so no compilation is required. But on CentOS 7, we need to make sure all the tools and libraries needed for compiling are in place first.

具体解决步骤 #

Here's what you can try, step by step:

1. Install essential compilation dependencies
   Twisted requires C compilers and Python development libraries to build from source. Run this command with sudo to install them:

   sudo yum install gcc python36-devel openssl-devel libffi-devel
   

   These packages provide all the core tools needed to compile Twisted without hitting missing library errors.

2. Upgrade pip to the latest version
   Outdated pip versions might not handle binary packages or source compilation smoothly. Upgrade pip inside your virtual environment first:

   pip install --upgrade pip
   

3. Force pip to use binary packages (if available)
   Try installing Twisted with the --only-binary flag to tell pip to prioritize pre-built wheels over source archives (if compatible builds exist for your system):

   pip install --only-binary :all: Twisted>=13.1.0
   

   If a matching wheel exists, this will skip the compilation step entirely and fix the hang immediately.

4. Re-attempt installing Scrapy
   Once dependencies are installed and pip is updated, go back to your original command:

   pip install scrapy
   

   This time, Twisted should either compile successfully (using the dependencies you installed) or use a binary package, avoiding the hang.

如果仍然卡住怎么办？ #

If compilation is taking an unusually long time, it might just be slow on your system—give it 5-10 minutes to finish. If it truly hangs indefinitely, you can manually compile Twisted with verbose output to see exactly where it's stuck:

# Download the source archive pip was trying to use
wget https://files.pythonhosted.org/packages/source/T/Twisted/Twisted-17.9.0.tar.bz2
# Extract it
tar xvjf Twisted-17.9.0.tar.bz2
cd Twisted-17.9.0
# Run build with verbose logging
python setup.py build -v

The verbose output will show every step of the compilation process, making it easy to spot where it's getting stuck.

内容的提问来源于stack exchange，提问作者cem akbulut