多进程函数无限运行问题：如何充分利用12核CPU？

问题背景 #

1. 爬虫场景 #

单进程爬虫可正常运行，但12核CPU利用率极低，改用多进程后代码陷入无限运行状态。原爬虫代码与多进程调用代码如下：

原爬虫函数：

daily_data=[]
def scrape_article(i,article_url):
    try:
        article_html=requests.get(article_url).content
        article_soup=BeautifulSoup(article_html, 'html.parser')

        article_id=article_soup.select('span[class*="sc-6e54cb25-12 GRQvX"]')
        article_title=article_soup.select_one('div[class="screen-section"] div[class*="sc-"] h2[class*="jvaJJV"]')
        date_published=article_soup.select('span[class*="sc-6e54cb25-12 GRQvX"]')
        appatment_price=article_soup.select_one('aside[class="detail-page-aside"] h1[id="price"]')
        appatment_address=article_soup.select_one('div[class="screen-section"] a[id="address"]')
        apparment_details_1=set([i.get_text(separator='|') if i else None for i in article_soup.select('div[id="details_desc"] div[class*="sc-5fa917ee-0"] div')])
        apparment_details_2=set([i.get_text(separator='|') if i else None for i in article_soup.select('div[id="details_desc"] div[class*="sc-1b705347-0"] div')])

        article_id=article_id[-1].text if article_id else None
        article_title=article_title.text if article_title else None
        date_published=date_published[-2].text if date_published else None
        appatment_price=appatment_price.text if appatment_price else None
        appatment_address=appatment_address.text if appatment_address else None

        aparment_data_dict={'Article ID':article_id,'Article URL':article_url,'Title':article_title,'Date Published':date_published,'Price':appatment_price,'Address':appatment_address,'Details_1':apparment_details_1,'Details_2':apparment_details_2}
        daily_data.append(aparment_data_dict)
    except:
        pass

原多进程调用代码：

import multiprocessing
if __name__ == "__main__":
    daily_article_urls = daily_article_urls# Define your list of article URLs

    with multiprocessing.Pool(processes=12) as pool:
        pool.map(scrape_article, enumerate(daily_article_urls))
    pool.close()
    pool.join()

2. 计算函数场景 #

采用相同多进程调用方式，运行简单数学计算函数时同样出现异常，代码如下：

import multiprocessing
import math

def calculate(number):
    try:
        calculated_number = math.sqrt((number**2 + 5*number**3 + 1022))
        return calculated_number
    except Exception as e:
        return None

if __name__ == "__main__":
    numbers = list(range(10))  # Define your list of article URLs

    with multiprocessing.Pool(processes=12) as pool:
        results = pool.map(calculate, numbers)
    for result in results:
        print(result)

问题原因与修复方案 #

一、爬虫多进程问题修复 #

核心问题

1. 参数传递不匹配：scrape_article需要两个参数，但pool.map仅能传递单个参数，enumerate返回的元组被整体传入函数，引发未捕获异常（原函数用except:吞掉所有错误），导致进程卡住。
2. 全局变量无法跨进程共享：多进程中每个子进程拥有独立内存空间，修改全局daily_data无法同步到主进程，数据收集失效。
3. 无请求超时机制：网络请求可能因异常挂起，导致进程无限等待。

修改后的完整代码

import requests
from bs4 import BeautifulSoup
import multiprocessing

def scrape_article(args):
    i, article_url = args  # 解包元组参数
    try:
        # 添加10秒超时，避免请求挂起
        article_html = requests.get(article_url, timeout=10).content
        article_soup = BeautifulSoup(article_html, 'html.parser')

        article_id = article_soup.select('span[class*="sc-6e54cb25-12 GRQvX"]')
        article_title = article_soup.select_one('div[class="screen-section"] div[class*="sc-"] h2[class*="jvaJJV"]')
        date_published = article_soup.select('span[class*="sc-6e54cb25-12 GRQvX"]')
        appatment_price = article_soup.select_one('aside[class="detail-page-aside"] h1[id="price"]')
        appatment_address = article_soup.select_one('div[class="screen-section"] a[id="address"]')
        apparment_details_1 = set([i.get_text(separator='|') if i else None for i in article_soup.select('div[id="details_desc"] div[class*="sc-5fa917ee-0"] div')])
        apparment_details_2 = set([i.get_text(separator='|') if i else None for i in article_soup.select('div[id="details_desc"] div[class*="sc-1b705347-0"] div')])

        article_id = article_id[-1].text if article_id else None
        article_title = article_title.text if article_title else None
        date_published = date_published[-2].text if date_published else None
        appatment_price = appatment_price.text if appatment_price else None
        appatment_address = appatment_address.text if appatment_address else None

        # 返回数据，由主进程统一收集
        return {'Article ID': article_id, 'Article URL': article_url, 'Title': article_title, 
                'Date Published': date_published, 'Price': appatment_price, 'Address': appatment_address, 
                'Details_1': apparment_details_1, 'Details_2': apparment_details_2}
    except Exception as e:
        # 打印异常便于排查，不吞掉错误
        print(f"处理URL {article_url} 出错: {str(e)}")
        return None

if __name__ == "__main__":
    daily_article_urls = []  # 替换为你的实际URL列表
    # 自动适配CPU核心数，无需手动指定
    with multiprocessing.Pool(processes=multiprocessing.cpu_count()) as pool:
        daily_data = pool.map(scrape_article, enumerate(daily_article_urls))
    # 过滤无效数据
    daily_data = [item for item in daily_data if item is not None]
    print(f"共爬取有效数据 {len(daily_data)} 条")

关键修改点

• 调整函数参数，接收元组并解包，适配pool.map的参数规则；
• 给requests.get添加timeout，避免网络请求无限挂起；
• 函数返回数据，主进程统一收集，放弃全局变量跨进程共享；
• 捕获并打印异常，方便排查问题；
• 用multiprocessing.cpu_count()自动获取核心数，适配不同硬件。

二、计算函数多进程问题修复 #

核心问题

原代码未打印异常信息，无法排查错误；进程数设置超过任务数（10个任务开12个进程），造成资源浪费；Windows环境下多进程启动可能因导入机制出现隐性问题。

修改后的完整代码

import multiprocessing
import math

def calculate(number):
    try:
        calculated_number = math.sqrt((number**2 + 5*number**3 + 1022))
        return calculated_number
    except Exception as e:
        print(f"计算数字 {number} 出错: {str(e)}")
        return None

if __name__ == "__main__":
    numbers = list(range(10))
    # 进程数设为核心数与任务数的较小值，避免资源浪费
    process_count = min(multiprocessing.cpu_count(), len(numbers))
    with multiprocessing.Pool(processes=process_count) as pool:
        results = pool.map(calculate, numbers)
    # 关联原数字输出，结果更清晰
    for idx, result in enumerate(results):
        print(f"数字 {numbers[idx]} 的计算结果: {result}")

关键修改点

• 打印异常信息，便于定位错误；
• 动态设置进程数，避免创建多余进程；
• 输出时关联原数字，提升结果可读性。

内容的提问来源于stack exchange，提问作者beridzeg45