Dataflow Flex Template启动触发双Job：一失败一成功的问题求助

运行命令 #

gcloud dataflow flex-template run $DF_JOB_NAME --template-file-gcs-location $FLEX_SPEC_PATH --region $REGION --service-account-email $SERVICE_ACCOUNT_EMAIL

问题现象 #

已成功构建并上传Flex Template镜像和Spec，任务可正常执行，但每次启动会触发两个独立Dataflow Job：

• 第一个Job用于构建任务运行的Docker容器，最终因轮询超时失败
• 第二个Job为实际执行ETL任务的Job，可成功完成

相关配置 #

Dockerfile #

FROM gcr.io/dataflow-templates-base/python39-template-launcher-base 

RUN apt-get update
# Upgrade pip and install the requirements.
RUN pip install --no-cache-dir --upgrade pip
RUN pip install apache-beam[gcp]
RUN pip install google-cloud-secret-manager==2.16.0

RUN mkdir -p /dataflow/template
RUN mkdir -p /dataflow/src
WORKDIR /dataflow/template

COPY src/test_job_name.py /dataflow/template/test_job_name.py
COPY src/test_job_name.json /dataflow/template/test_job_name.json
COPY __init__.py /dataflow/template/__init__.py
COPY setup.py /dataflow/template/setup.py

ENV FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE=""
ENV FLEX_TEMPLATE_PYTHON_PY_OPTIONS=""
ENV FLEX_TEMPLATE_PYTHON_EXTRA_PACKAGES=""
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="/dataflow/template/run_test_job_name.py"
ENV FLEX_TEMPLATE_PYTHON_SETUP_FILE="/dataflow/template/setup.py"

# Since we already downloaded all the dependencies, there's no need to rebuild everything.
ENV PIP_NO_DEPS=True

Python主文件参数解析 #

parser = argparse.ArgumentParser()
parser.add_argument("-ds", "--date")
parser.add_argument("-p", "--project")
parser.add_argument("-t", "--temp_location")
parser.add_argument("-s", "--staging_location")
parser.add_argument("-r", "--runner")
parser.add_argument("-se", "--setup")
parser.add_argument("-sa", "--service_account_email")
parser.add_argument("-re", "--region")
args, additional_dataflow_args = parser.parse_known_args()

PipelineOptions配置 #

p_options = {
    "project": args.project,
    "staging_location": args.staging_location,
    "temp_location": args.temp_location,
    "runner": args.runner,
    "setup_file": args.setup,
    "region": args.region,
    "save_main_session": True,
    "service_account_email": args.service_account_email,
}

已尝试的解决方法 #

• 确保仅传递一次镜像
• 移除Python文件中的job_name选项

以上操作均未解决问题，寻求可行的解决办法。

内容的提问来源于stack exchange，提问作者snark17