GitLab Runner配置：YML文件从外部源取数生成脚本的可行性

Great question! GitLab CI/CD's .gitlab-ci.yml is statically parsed by the runner before any jobs execute, so you can't directly write a loop in the YAML to pull data from MySQL and generate content on the fly. But there are two reliable workarounds to get exactly the behavior you want:

方法1：在单个部署任务内动态执行循环逻辑 #

If you just need to run deployment commands for multiple targets (from your MySQL data) within a single CI job, this is the simplest approach. You'll fetch the data during the job's script phase, then loop through it with shell commands.

Here's a sample config:

deploy_dev:
  stage: deploy
  # Use an image that has MySQL client and SSH tools (or install them below)
  image: alpine:latest
  before_script:
    # Install required tools if your base image doesn't have them
    - apk add --no-cache mysql-client openssh-client ssh-agent
    # Load your SSH key (store the key as a masked CI/CD variable for security)
    - echo "$SSH_PRIVATE_KEY" > ~/.ssh/key.pem
    - chmod 600 ~/.ssh/key.pem
  script:
    - echo "Fetching deployment targets from MySQL..."
    # Pull data from MySQL - use CI/CD variables for DB credentials!
    - DEPLOY_TARGETS=$(mysql -h "$DB_HOST" -u "$DB_USER" -p"$DB_PASSWORD" -D "$DB_NAME" -sN -e "SELECT server_ip FROM deploy_servers WHERE environment='dev'")
    # Loop through each target and run deployment steps
    - for server_ip in $DEPLOY_TARGETS; do
        echo "Starting deployment to $server_ip";
        eval $(ssh-agent);
        ssh-add ~/.ssh/key.pem;
        # Replace with your actual deployment commands
        ssh -p22 root@$server_ip "git pull && restart-app-service";
      done
  only:
    - main # Run this job only when pushing to the main branch

Key notes here: Store all sensitive values (DB credentials, SSH key) as masked, protected CI/CD variables in your GitLab project settings—never hardcode them!

方法2：动态生成完整的.gitlab-ci.yml配置 #

If you want to create separate CI jobs for each deployment target (e.g. deploy_dev_server1, deploy_dev_server2), you can generate the YAML config dynamically in a pre-job, then have GitLab Runner use that generated config.

Step 1: Create a config generation script #

Write a script (e.g. Python, Bash) that connects to MySQL and outputs a complete .gitlab-ci.yml file. Here's a Python example:

import mysql.connector
import os

# Fetch DB credentials from GitLab CI variables
db_config = {
    "host": os.getenv("DB_HOST"),
    "user": os.getenv("DB_USER"),
    "password": os.getenv("DB_PASSWORD"),
    "database": os.getenv("DB_NAME")
}

# Connect to MySQL and fetch deployment targets
db = mysql.connector.connect(**db_config)
cursor = db.cursor()
cursor.execute("SELECT server_name, server_ip FROM deploy_servers WHERE environment='dev'")
servers = cursor.fetchall()

# Build the dynamic CI config
ci_config = """
stages:
  - deploy
"""

for server_name, server_ip in servers:
    # Add a unique job for each server
    ci_config += f"""
deploy_dev_{server_name}:
  stage: deploy
  script:
    - echo "Deploying to {server_name} ({server_ip})"
    - eval $(ssh-agent)
    - ssh-add ~/.ssh/key.pem
    - ssh -p22 root@{server_ip} "git pull && restart-app-service"
  only:
    - main
"""

# Save the generated config to a file
with open(".gitlab-ci-dynamic.yml", "w") as f:
    f.write(ci_config)

db.close()

Step 2: Base .gitlab-ci.yml to trigger the generation #

This base config runs the generation script first, then includes the dynamically created config:

generate_ci_config:
  stage: prepare
  image: python:3.10
  before_script:
    - pip install mysql-connector-python
    # Prepare SSH key (from CI/CD variable)
    - echo "$SSH_PRIVATE_KEY" > ~/.ssh/key.pem
    - chmod 600 ~/.ssh/key.pem
  script:
    - python generate_ci_config.py
  artifacts:
    paths:
      - .gitlab-ci-dynamic.yml
  only:
    - main

# Include the dynamically generated config
include:
  - artifact: .gitlab-ci-dynamic.yml
    job: generate_ci_config

When you push code, GitLab will first run the generate_ci_config job to create the dynamic YAML, then execute all the deployment jobs defined in that file.

Critical Security & Practical Tips #

• Never hardcode secrets: Use GitLab's CI/CD Variables (masked/protected) for DB passwords, SSH keys, and other sensitive data.
• Network access: Ensure your GitLab Runner has network access to your MySQL database (add the Runner's IP to your DB's whitelist).
• Image compatibility: Use CI images that include the tools you need, or install them in before_script to avoid runtime errors.

内容的提问来源于stack exchange，提问作者user3836484