Docker管理的Laravel集成Vault后偶发数据库连接拒绝问题排查

问题背景 #

原本使用Helm Chart管理环境凭证，集成Vault后应用偶尔抛出「db refused to connect error」错误。已在Dockerfile中创建.env文件，通过service-entry.sh更新该文件，且已验证.env生成内容正确。相关配置代码如下：

Dockerfile配置 #

FROM php:7.3-fpm

# Set working directory
WORKDIR /b2b

# Install dependencies
RUN apt-get update && apt-get install -y apt-utils
RUN apt-get install -y \
   wget \
   build-essential \
   libpng-dev \
   libjpeg62-turbo-dev \
   libfreetype6-dev \
   locales \
   zip \
   libzip-dev \
   jpegoptim optipng pngquant gifsicle \
   vim \
   unzip \
   curl \
   libxml2-dev \
   openssl \
   procps \
   jq

# Clear cache
RUN apt-get clean && rm -rf /var/lib/apt/lists/*

# Install extensions
RUN docker-php-ext-install pdo_mysql mbstring zip exif pcntl soap
RUN docker-php-ext-configure gd --with-gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ --with-png-dir=/usr/include/
RUN docker-php-ext-install gd
RUN docker-php-ext-install bcmath

# Install composer
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer

# Copy existing application directory contents
COPY service-entry.sh .

COPY ./aqualens_b2b_in .

# Create .env file and set permissions
RUN touch .env && chmod 0777 .env

RUN chmod -R 0777 ./storage
RUN chmod -R 0777 ./bootstrap

RUN mkdir -p ./storage/framework/cache
RUN mkdir -p ./storage/framework/sessions
RUN mkdir -p ./storage/framework/views

RUN chmod -R 0777 ./storage

# Set the entry point for the container
CMD ["sh","/b2b/service-entry.sh" ]

RUN composer install
RUN php artisan optimize

service-entry.sh脚本 #

#!/bin/sh

# For VAULT

: ${VAULT_ENABLED:="true"}
: ${VAULT_SERVICE_ACCOUNT_TOKEN_FILE:="/var/run/secrets/kubernetes.io/serviceaccount/token"}
: ${VAULT_URI:="http://vault-cluster-url.local:2000"}
: ${VAULT_ROLE:="vault-myapp-config-b2b-auth-role"}
: ${VAULT_BACKEND:="company/myapp/myapp-config"}
: ${VAULT_DEFAULT_CONTEXT:="myapp-b2b"}

# Check and print the current directory and its permissions
echo "Current directory: $(pwd)"
ls -ld .
ls -a
ls -l

if [ "$VAULT_ENABLED" = "true" ]; then
    kubernetesToken=$(cat "${VAULT_SERVICE_ACCOUNT_TOKEN_FILE}")
    if [ $? -ne 0 ]; then
        echo "Error: Unable to read Kubernetes service account token file"
    fi

    vaultEndpoint="${VAULT_URI}/v1/auth/kubernetes/login"
    H1_KUBE="Content-Type: application/json"
    H2_KUBE="Authorization: Bearer ${kubernetesToken}"

    requestData="{\"role\":\"${VAULT_ROLE}\",\"jwt\":\"${kubernetesToken}\"}"
    response=$(curl -s -X POST -H "$H1_KUBE" -H "$H2_KUBE" --data "${requestData}" "${vaultEndpoint}")

    if [ $? -ne 0 ]; then
        echo "Error: Unable to authenticate with Vault"
    fi

    access_token=$(echo "${response}" | jq -r .auth.client_token)
    if [ -z "$access_token" ] || [ "$access_token" = "null" ]; then
        echo "Error: Unable to retrieve access token from Vault response"
    fi

    H1_VAULT="X-Vault-Token: ${access_token}"
    H2_VAULT="Accept: application/json"

    RESPONSE_VAULT=$(curl -s -H "$H1_VAULT" -H "$H2_VAULT" "${VAULT_URI}/v1/${VAULT_BACKEND}/${VAULT_DEFAULT_CONTEXT}")

    if [ $? -ne 0 ]; then
        echo "Error: Unable to fetch vault data"
    fi

    export_statements=$(echo "$RESPONSE_VAULT" | jq -r '.data | to_entries[] | @sh "export \(.key)=\(.value)"')
    if [ -z "$export_statements" ]; then
        echo "Error: No data retrieved from Vault"
    fi

    echo "Logging Qualens Vault Integration:"
    echo "export_statements"
    echo "$export_statements"

    # Read each export statement and process it
    echo "$export_statements" | while IFS= read -r line; do
        eval "$line"
        key=$(echo "$line" | cut -d'=' -f1 | cut -d' ' -f2)
        value=$(echo "$line" | cut -d'=' -f2- | sed "s/^'//" | sed "s/'$//")
        if grep -q "^${key}=" .env; then
            sed -i "s|^${key}=.*|${key}=${value}|g" .env
        else
            echo "${key}=${value}" >> .env
        fi
    done

    echo "Contents of .env file after update:"
    cat .env

fi

# For PHP
php-fpm

排查建议 #

• 给Vault操作添加重试机制：当前脚本中Vault认证、配置拉取的curl请求无重试逻辑，网络波动或Vault临时不可用会导致配置拉取失败。给curl命令加上--retry 3 --retry-delay 2参数，允许重试3次，每次间隔2秒。
• 验证PHP-FPM启动时机：脚本更新完.env后直接启动php-fpm，可能存在文件写入未完成就启动服务的情况。在启动前添加sleep 1延迟，或检查.env中关键数据库配置项是否存在后再启动。
• 完善错误退出逻辑：当前脚本在Vault操作失败时仅打印错误，但仍会启动php-fpm，导致应用使用无效配置。在获取token失败、拉取配置失败等关键步骤添加exit 1，避免服务带病启动。
• 检查容器时间同步：K8s服务账户token和Vault认证依赖时间同步，若容器时间与集群偏差较大，会导致token验证失败。在Dockerfile中安装ntp服务，或挂载host的时间配置。
• 监控Vault操作耗时：添加日志记录Vault认证、配置拉取的耗时，排查偶发超时情况。比如给curl命令加上-w "%{time_total}\n"打印请求耗时，或记录时间戳对比。
• 确认环境变量生效范围：脚本中eval导出的变量仅在当前shell生效，php-fpm作为子进程可能无法继承。启动php-fpm前明确导出所有Vault获取的变量，或确保Laravel优先读取.env文件。
• 调整数据库连接配置：偶发连接拒绝可能是数据库端临时负载过高，在Laravel的config/database.php中增加连接超时和重试设置：

  'options' => [
      PDO::ATTR_TIMEOUT => 5, // 连接超时5秒
      PDO::ATTR_PERSISTENT => true, // 启用持久连接
  ]
  

• 排查容器并发启动冲突：多容器同时启动可能触发Vault限流，查看Vault日志是否有相关记录，或调整Vault的Kubernetes auth后端并发设置。

内容的提问来源于stack exchange，提问作者Shakir Baba