使用apache/beam_python3.9_sdk:2.40.0的Dataflow任务遇反序列化错误求解决方案

问题重现 #

在Google Cloud Dataflow Python环境部署自定义容器时，执行以下WordCount示例命令会触发序列化报错：

python -m apache_beam.examples.wordcount \
--output gs://<your-output-dir> \
--runner=DataflowRunner \
--project=<your-project-id> \
--region us-central1 \
--temp_location=gs://<your-tmp-location> \
--worker_harness_container_image=apache/beam_python3.9_sdk:2.40.0 \
--experiment=use_runner_v2

Worker端报错日志：

Error message from worker: Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py", line 285, in loads
    return dill.loads(s)
  File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 275, in loads
    return load(file, ignore, **kwds)
  File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 270, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
  File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 472, in load
    obj = StockUnpickler.load(self)
  File "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 462, in find_class
    return StockUnpickler.find_class(self, module, name)
AttributeError: Can't get attribute '_create_code' on <module 'dill._dill' from '/usr/local/lib/python3.9/site-packages/dill/_dill.py'>

临时降级方案：使用apache/beam_python3.9_sdk:2.38.0容器可正常执行任务，但并非长期理想方案。

2.40.0版本的解决办法 #

1. 锁定dill版本到0.3.4 #

报错根源是Beam 2.40.0内置的dill 0.3.5.1存在序列化兼容性问题，在自定义容器中强制降级dill版本：

• 在requirements.txt中添加：

  dill==0.3.4
  

• 或在Dockerfile中执行：

  RUN pip install dill==0.3.4 --force-reinstall
  

2. 关闭多SDK容器实验参数 #

Runner V2的use_multiple_sdk_containers实验特性可能加剧序列化冲突，添加参数禁用该特性：

--experiment=no_use_multiple_sdk_containers

修改后的完整命令：

python -m apache_beam.examples.wordcount \
--output gs://<your-output-dir> \
--runner=DataflowRunner \
--project=<your-project-id> \
--region us-central1 \
--temp_location=gs://<your-tmp-location> \
--worker_harness_container_image=apache/beam_python3.9_sdk:2.40.0 \
--experiment=use_runner_v2 \
--experiment=no_use_multiple_sdk_containers

3. 升级到Beam 2.41.0+版本 #

该序列化问题已在Beam 2.41.0及后续版本中修复，直接使用对应SDK容器即可：

--worker_harness_container_image=apache/beam_python3.9_sdk:2.41.0

内容的提问来源于stack exchange，提问作者johnmcs