使用Python多进程池处理像素时遇冻结模块警告及脚本循环问题

问题概述 #

原串行遍历像素实现网游自动化，因速度瓶颈改用multiprocessing.Pool并行处理后，出现两个核心问题：

• 调试器输出冻结模块警告
• 脚本不断在Chrome中打开新标签页

错误信息 #

0.02s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

原代码 #

GetPixelData函数 #

def GetPixelData(x_pos, x_size, y_size, pixel_number, pixels_obstacle, pixels_coffee):
    global coffee_amount

    x = pixel_number / x_size
    y = pixel_number % y_size

    if pixels_obstacle.getpixel((x, y))[2] < 240:
        Jump(driver, canvas)
        print("Object")
    elif pixels_coffee.getpixel((x, y))[1] < 50:
        Jump(driver, canvas)
        coffee_amount += 1
        print("+1 Coffee (" + str(coffee_amount) + " total)")
    
    return x, y

并行调用代码 #

# Get pixels in bounds
bounds_obstacle = (x_pos, 830, x_pos + x_size, 830 + y_size) * screen_multiplier
bounds_coffee = (x_pos, 460, x_pos + x_size, 460 + y_size) * screen_multiplier
pixels_obstacle = ImageGrab.grab(bbox=bounds_obstacle)
pixels_coffee = ImageGrab.grab(bbox=bounds_coffee)

# Process pixels in parallel
args = [x_pos, x_size, y_size, pixels_obstacle, pixels_coffee]
iterable = ([args, pixel_number] for pixel_number in range(x_size * y_size))
pixel_pool = pool.Pool(os.cpu_count())
for r in pixel_pool.imap_unordered(GetPixelData, iterable):
    print(str(r))

问题分析 #

1. 冻结模块警告：Python调试器与frozen模块冲突，属于调试环境提示，不影响脚本核心运行，但会干扰断点调试。
2. Chrome重复启动：Windows下多进程采用spawn模式，子进程会重新执行整个脚本，若Chrome驱动初始化代码未隔离，每个子进程都会创建新的Chrome实例。
3. 参数传递错误：原代码中iterable的元素结构与函数参数不匹配，imap_unordered会将每个元素作为单个参数传入，导致函数参数缺失。
4. 像素坐标错误：x = pixel_number / x_size得到浮点数，而getpixel需要整数坐标，会触发隐式转换错误或逻辑异常。
5. WebDriver安全问题：WebDriver实例不支持多进程并发操作，多个进程同时调用Jump会导致浏览器行为异常。
6. 全局变量共享失效：多进程中全局变量coffee_amount无法跨进程同步，每个子进程的修改不会反映到主进程。

解决方案 #

1. 消除冻结模块警告 #

运行脚本时添加参数：

python -Xfrozen_modules=off your_script.py

或设置环境变量（永久生效）：

# Windows
set PYDEVD_DISABLE_FILE_VALIDATION=1
# Linux/macOS
export PYDEVD_DISABLE_FILE_VALIDATION=1

2. 隔离Chrome驱动初始化代码 #

将驱动初始化、浏览器启动等代码放入if __name__ == '__main__':块，避免子进程重复执行：

if __name__ == '__main__':
    driver = webdriver.Chrome()  # 你的Chrome驱动初始化
    canvas = driver.find_element(...)  # 页面元素定位
    # 其他全局初始化代码

3. 修正多进程参数传递 #

使用starmap替代imap_unordered，将每个任务的参数打包为元组，确保函数能正确接收参数：

iterable = [(x_size, y_size, pn, pixels_obstacle, pixels_coffee) 
            for pn in range(x_size * y_size)]

4. 修复像素坐标计算 #

用整数除法//替代浮点数除法/，确保坐标为整数：

x = pixel_number // x_size
y = pixel_number % y_size

5. 分离像素检测与浏览器操作 #

子进程仅返回检测结果，主进程统一执行Jump操作，避免多进程操作WebDriver：

def GetPixelData(x_size, y_size, pixel_number, pixels_obstacle, pixels_coffee):
    x = pixel_number // x_size
    y = pixel_number % y_size
    # 障碍物检测
    if pixels_obstacle.getpixel((x, y))[2] < 240:
        return True, False
    # 咖啡检测
    if pixels_coffee.getpixel((x, y))[1] < 50:
        return True, True
    return False, False

6. 跨进程共享状态 #

使用multiprocessing.Value创建共享变量，记录咖啡数量，修改时加锁避免竞争：

from multiprocessing import Value

if __name__ == '__main__':
    coffee_amount = Value('i', 0)  # 'i'表示整数类型
    # ...其他代码
    # 主进程处理结果
    for need_jump, is_coffee in results:
        if need_jump:
            Jump(driver, canvas)
            if is_coffee:
                with coffee_amount.get_lock():
                    coffee_amount.value += 1
                print(f"+1 Coffee ({coffee_amount.value} total)")
            else:
                print("Object")

修正后的完整代码示例 #

import multiprocessing as mp
from multiprocessing import Value
from PIL import ImageGrab
from selenium import webdriver

def GetPixelData(x_size, y_size, pixel_number, pixels_obstacle, pixels_coffee):
    x = pixel_number // x_size
    y = pixel_number % y_size

    obstacle_pixel = pixels_obstacle.getpixel((x, y))
    if obstacle_pixel[2] < 240:
        return True, False
    
    coffee_pixel = pixels_coffee.getpixel((x, y))
    if coffee_pixel[1] < 50:
        return True, True
    
    return False, False

def Jump(driver, canvas):
    # 你的跳跃逻辑，例如模拟点击或按键
    canvas.click()

if __name__ == '__main__':
    # 初始化浏览器
    driver = webdriver.Chrome()
    driver.get("你的游戏页面URL")
    canvas = driver.find_element("css selector", "你的canvas选择器")
    
    # 配置参数
    screen_multiplier = 1
    x_pos = 100
    x_size = 40
    y_size = 10
    
    # 共享咖啡计数
    coffee_amount = Value('i', 0)
    
    # 获取像素区域
    bounds_obstacle = (x_pos, 830, x_pos + x_size, 830 + y_size) * screen_multiplier
    bounds_coffee = (x_pos, 460, x_pos + x_size, 460 + y_size) * screen_multiplier
    pixels_obstacle = ImageGrab.grab(bbox=bounds_obstacle)
    pixels_coffee = ImageGrab.grab(bbox=bounds_coffee)
    
    # 准备并行任务参数
    iterable = [(x_size, y_size, pn, pixels_obstacle, pixels_coffee) 
                for pn in range(x_size * y_size)]
    
    # 并行处理像素
    with mp.Pool(mp.cpu_count()) as pixel_pool:
        results = pixel_pool.starmap(GetPixelData, iterable)
    
    # 主进程处理操作指令
    for need_jump, is_coffee in results:
        if need_jump:
            Jump(driver, canvas)
            if is_coffee:
                with coffee_amount.get_lock():
                    coffee_amount.value += 1
                print(f"+1 Coffee ({coffee_amount.value} total)")
            else:
                print("Object")
    
    driver.quit()

内容的提问来源于stack exchange，提问作者guiguig