Python中传递Selenium实例至新进程出现异常问题

报错信息 #

Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\tkinter\__init__.py", line 1892, in __call__
    return self.func(*args)
  File "c:\Users\user\Desktop\coding\PokeClicker Bot\Version 3 (Gui) (WIP)\version3.py", line 37, in <lambda>
    item = tk.Button(root, text='Run', relief='raised', command=lambda:toggle(item))
  File "c:\Users\user\Desktop\coding\PokeClicker Bot\Version 3 (Gui) (WIP)\version3.py", line 54, in toggle
    startRun()
  File "c:\Users\user\Desktop\coding\PokeClicker Bot\Version 3 (Gui) (WIP)\version3.py", line 62, in startRun
    p.start()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_io.TextIOWrapper' object
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

精简代码 #

def init():
    global ffD
    ffD = WD.Firefox(os.getcwd(), options=options)
    ffD.get('https://www.pokeclicker.com/')
    root.mainloop()
    
def runThroughDungeon(driver): #Currently just driver.close() to test.
    driver.close()

def startRun(): #Called from toggling tkinter button.
    global p
    p = mp.Process(target = runThroughDungeon, args=(ffD,))
    p.start()

def stopRun():
    p.kill()

if __name__ == '__main__':
    init()

问题原因 #

Selenium的WebDriver实例无法通过Python的序列化（pickle）机制传递给子进程。因为WebDriver内部包含大量无法被序列化的资源，比如_io.TextIOWrapper文件对象、浏览器进程的底层连接句柄等。Windows系统下的multiprocessing默认使用spawn启动方式，会重新初始化Python解释器，要求所有传递给子进程的参数必须能被序列化，因此直接传递Driver实例会触发报错。

解决方案 #

方案1：在子进程内初始化WebDriver #

不再在主进程创建Driver实例，而是让子进程自行启动浏览器、执行任务并关闭，避免跨进程传递无法序列化的对象：

def runThroughDungeon():
    # 子进程内初始化Driver
    ffD = WD.Firefox(os.getcwd(), options=options)
    ffD.get('https://www.pokeclicker.com/')
    # 执行你的业务逻辑
    ffD.close()

def startRun():
    global p
    p = mp.Process(target=runThroughDungeon)
    p.start()

方案2：改用线程替代多进程 #

如果需要复用主进程的Driver实例，可以使用threading模块。线程共享主进程的内存空间，不需要序列化对象，直接传递Driver实例即可。注意：tkinter的UI操作必须在主线程执行，但Selenium的操作可以在子线程中进行：

import threading

def runThroughDungeon(driver):
    driver.close()

def startRun():
    global p
    p = threading.Thread(target=runThroughDungeon, args=(ffD,))
    p.start()

内容的提问来源于stack exchange，提问作者cheis