GitHub Actions中Django并行测试报错：无法pickle traceback对象

问题描述 #

我在GitHub Action的yml文件里配置了Django测试步骤：

python manage.py test --failfast --parallel 2

加上--parallel 2参数后测试直接失败，报错信息如下：

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/django/test/runner.py", line 449, in _run_subsuite
result = runner.run(subsuite)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/django/test/runner.py", line 366, in run
test(result)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/unittest/suite.py", line 84, in call
return self.run(*args, **kwds)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/unittest/suite.py", line 122, in run
test(result)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/django/test/testcases.py", line 381, in call
self._setup_and_call(result)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/django/test/testcases.py", line 416, in _setup_and_call
super().call(result)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/unittest/case.py", line 650, in call
return self.run(*args, **kwds)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/unittest/case.py", line 599, in run
self._feedErrorsToResult(result, outcome.errors)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/unittest/case.py", line 516, in _feedErrorsToResult
result.addFailure(test, exc_info)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/django/test/runner.py", line 296, in addFailure
self.check_picklable(test, err)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/django/test/runner.py", line 216, in check_picklable
self._confirm_picklable(err)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/django/test/runner.py", line 186, in _confirm_picklable
pickle.loads(pickle.dumps(obj))
TypeError: cannot pickle 'traceback' object
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/runner/work/fiscozen/fiscozen/fiscozen_django/manage.py", line 24, in
execute_from_command_line(sys.argv)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/django/core/management/init.py", line 442, in execute_from_command_line
utility.execute()
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/django/core/management/init.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/django/core/management/commands/test.py", line 24, in run_from_argv
super().run_from_argv(argv)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/django/core/management/base.py", line 412, in run_from_argv
self.execute(*args, **cmd_options)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/django/core/management/base.py", line 458, in execute
output = self.handle(*args, **options)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/django/core/management/commands/test.py", line 68, in handle
failures = test_runner.run_tests(test_labels)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/django/test/runner.py", line 1061, in run_tests
result = self.run_suite(suite)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/django/test/runner.py", line 983, in run_suite
return runner.run(suite)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/unittest/runner.py", line 184, in run
test(result)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/unittest/suite.py", line 84, in call
return self.run(*args, **kwds)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/django/test/runner.py", line 534, in run
subsuite_index, events = test_results.next(timeout=0.1)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/multiprocessing/pool.py", line 873, in next
raise value
TypeError: cannot pickle 'traceback' object
Error: Process completed with exit code 1.

想问有没有其他开发者遇到过这个问题？

问题原因与解决办法 #

这个问题不少开发者都碰到过，核心原因是Django并行测试时，子进程抛出异常后需要把错误信息序列化（pickle）传递给主进程，但traceback对象本身无法被pickle序列化，导致报错。

常见的解决思路：

• 临时禁用并行：如果当下不需要并行提速，直接去掉--parallel 2参数，先保证测试正常运行，后续再深挖具体原因。
• 排查测试用例：检查你的测试用例里有没有自定义异常、类实例或者全局变量包含了无法pickle的对象（比如文件句柄、网络连接、存下来的traceback实例），这类对象会在并行模式下触发序列化失败。
• 升级Django版本：部分旧版Django在并行测试的错误处理上存在bug，升级到3.2+或4.x的稳定版本，大概率能修复这个问题。
• 自定义测试Runner：如果必须保留并行功能，可以自定义测试Runner，重写错误处理逻辑，把traceback转换成可序列化的字符串再传递，比如在check_picklable方法里将traceback对象转为文本信息，避免直接序列化原对象。

内容的提问来源于stack exchange，提问作者Luke