App Engine Flex Python3服务启动Dataflow作业失败：找不到可执行文件

我有一个已运行多年的App Engine Flex Python3服务，用于启动Dataflow作业。近一周所有作业均无法启动，且我未进行任何部署变更，推测是App Engine Flex内部发生变更导致。

错误日志 #

Executable ['/env/bin/python', 'setup.py', 'sdist', '--dist-dir', '/tmp/tmpmeiadu1k'] not found

Traceback (most recent call last):
  File "/env/lib/python3.7/site-packages/apache_beam/utils/processes.py", line 89, in check_output
    out = subprocess.check_output(*args, **kwargs)
  File "/opt/python3.7/lib/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
  File "/opt/python3.7/lib/python3.7/subprocess.py", line 488, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/opt/python3.7/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/opt/python3.7/lib/python3.7/subprocess.py", line 1482, in _execute_child
    restore_signals, start_new_session, preexec_fn)

BlockingIOError: [Errno 11] Resource temporarily unavailable

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/env/lib/python3.7/site-packages/flask/app.py", line 1523, in full_dispatch_request
    rv = self.dispatch_request()
  File "/env/lib/python3.7/site-packages/flask/app.py", line 1509, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/vmagent/app/src/utils/decorators.py", line 61, in decorated_function
    return f(*args, **kwargs)
  File "/home/vmagent/app/src/handlers/dataflow_task_handlers.py", line 179, in df_day_summary
    wait=False,
  File "/home/vmagent/app/dataflow_tasks/rec_day_summary.py", line 782, in run
    result = p.run()
  File "/env/lib/python3.7/site-packages/apache_beam/pipeline.py", line 550, in run
    self._options).run(False)
  File "/env/lib/python3.7/site-packages/apache_beam/pipeline.py", line 574, in run
    return self.runner.run_pipeline(self, self._options)
  File "/env/lib/python3.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 475, in run_pipeline
    artifacts = environments.python_sdk_dependencies(options)
  File "/env/lib/python3.7/site-packages/apache_beam/transforms/environments.py", line 806, in python_sdk_dependencies
    skip_prestaged_dependencies=skip_prestaged_dependencies)
  File "/env/lib/python3.7/site-packages/apache_beam/runners/portability/stager.py", line 282, in create_job_resources
    setup_options.setup_file, temp_dir, build_setup_args)
  File "/env/lib/python3.7/site-packages/apache_beam/runners/portability/stager.py", line 775, in _build_setup_package
    processes.check_output(build_setup_args)
  File "/env/lib/python3.7/site-packages/apache_beam/utils/processes.py", line 91, in check_output
    raise RuntimeError("Executable {} not found".format(args[0]))

RuntimeError: Executable ['/env/bin/python', 'setup.py', 'sdist', '--dist-dir', '/tmp/tmpmeiadu1k'] not found

相关配置说明 #

我的Pipeline选项中指定了setup_file为./setup.py，该文件位于项目根目录，内容如下：

import setuptools
    
REQUIRED_PACKAGES = [
   'google-cloud-error-reporting==1.1.2',
   'polyline==1.4.0',
]
setuptools.setup(
    name='dataflow_python_pipeline',
    version='1.0.0',
    description='DataFlow Python Pipeline',
    install_requires=REQUIRED_PACKAGES,
    packages=setuptools.find_packages(),
)

内容的提问来源于stack exchange，提问作者Alex