gcloud dataflow flex-template run中--temp-location参数不被识别问题排查

Let’s break down the likely reasons why this error only pops up in your Buildkite pipeline but works perfectly locally:

1. Outdated gcloud SDK Version in the Docker Image #

This is the most probable culprit. The --temp-location parameter for gcloud dataflow flex-template run was added in a specific newer version of the Google Cloud SDK. If your local gcloud installation is up-to-date but the cloud-sdk Docker image used in Buildkite is running an older release, it won’t recognize this parameter at all.

• How to verify: Add a line to your bash script to print the SDK version before running the Dataflow command:

  gcloud --version
  

  Compare this output with your local gcloud --version result. If the Docker image’s version is significantly behind, that’s your issue.
• Fix: Either pin your Docker image to a recent specific version (e.g., gcr.io/google.com/cloudsdktool/cloud-sdk:440.0.0 instead of relying on the default latest), or explicitly update the SDK inside the Dockerfile:

  FROM gcr.io/google.com/cloudsdktool/cloud-sdk
  RUN gcloud components update
  # Rest of your Dockerfile steps
  

2. Incorrect Parameter Ordering #

While your local run works, some versions of the gcloud SDK are strict about parameter placement. In your script, --temp-location is followed immediately by --parameters flags—older SDK versions might expect global flags (like --staging-location, --temp-location) to come before any --parameters or resource-specific flags.

• Fix: Reorder your command to group all global flags first, then --parameters, then remaining flags like --subnetwork:

  gcloud dataflow flex-template run ${JOB_NAME} \
      --template-file-gcs-location ${TEMPLATE_PATH}.json \
      --region us-central1 \
      --staging-location ${GCS_PATH}/staging/${JOB_NAME} \
      --temp-location ${GCS_PATH}/temp \
      --subnetwork=${SUBNETWORK} \
      --service-account-email=${SERVICE_ACCOUNT_EMAIL} \
      --parameters requirements_file=requirements.txt \
      --parameters input_subscription=${INPUT_SUBSCRIPTION} \
      --parameters output_table=${OUTPUT_TABLE} \
      --parameters subject=${SUBJECT} \
      --parameters schema_registry_url=${SCHEMA_REGISTRY_URL}
  

3. Environment Variable Expansion Issues in Buildkite #

It’s possible one of your environment variables (like ${GCS_PATH}) isn’t expanding correctly in the Buildkite pipeline, leading to malformed command arguments. For example, if ${GCS_PATH} is empty or contains unexpected whitespace, the CLI might misinterpret --temp-location as an invalid argument.

• How to check: Add debug logging to your script to print the full command before execution:

  echo "Executing command: gcloud dataflow flex-template run ${JOB_NAME} \
      --template-file-gcs-location ${TEMPLATE_PATH}.json \
      --region us-central1 \
      --staging-location ${GCS_PATH}/staging/${JOB_NAME} \
      --temp-location ${GCS_PATH}/temp \
      --parameters requirements_file=requirements.txt \
      --parameters input_subscription=${INPUT_SUBSCRIPTION} \
      --parameters output_table=${OUTPUT_TABLE} \
      --parameters subject=${SUBJECT} \
      --parameters schema_registry_url=${SCHEMA_REGISTRY_URL} \
      --subnetwork=${SUBNETWORK} \
      --service-account-email=${SERVICE_ACCOUNT_EMAIL}"
  

  Check the Buildkite logs to confirm all variables are replaced with their correct values.
• Fix: Ensure all required environment variables are properly set in your Buildkite pipeline configuration (either via the web UI or your buildkite.yml file).

4. Hidden Whitespace or Syntax Errors in the Script #

Sometimes invisible characters (like non-breaking spaces) or incorrect line continuation can cause the CLI to misparse arguments. Even if it works locally, line endings or whitespace might differ between your local environment and the Docker container running in Buildkite.

• Fix:
  • Use cat -A to inspect your bash script for hidden characters: cat -A gcloud-deploy-flex-template.sh
  • Double-check that every line ending with \ has no trailing whitespace after the backslash.

内容的提问来源于stack exchange，提问作者willwrighteng