使用dbt-athena-community+GitHub Actions执行工作流遇适配器错误

报错日志 #

dbt.exceptions.DbtProfileError: Runtime Error
Credentials in profile "default", target "dbt_dev" invalid: Runtime Error
Could not find adapter type athena!

本地基于AWS CLI配置的凭证可正常运行，但GitHub Actions工作流执行SQLFluff + dbt-athena-community时出现上述错误，核心问题为Actions环境缺少适配dbt-athena的必要配置。

项目目录结构 #

./.github/workflows/lint_models.yml
./models <----.sql files to lint
./.sqlfluff
./profiles.yml

相关配置文件 #

profiles.yml #

default:
  outputs:
    dbt_dev:
      database: awsdatacatalog
      region_name: eu-west-2
      s3_staging_dir: s3://athena-query-results-XXX/
      schema: dbt_test
      type: athena
      aws_profile_name: prof_name  <-- 已移除该行
      work_group: primary
  target: dbt_dev

.sqlfluff #

[sqlfluff]
templater = dbt
dialect = athena
sql_file_exts = .sql

[sqlfluff:templater:dbt]
project_dir = ./
profiles_dir = ./
profile = default  <-- 已移除该行
target = dbt_dev   <-- 已移除该行

lint_models.yml #

name: dbt sqlfluff
on:
  pull_request:
jobs:
  test-check:
    name: runner / sqlfluff (github-check)
    runs-on: ubuntu-latest
    steps:
      - name: install python
        uses: actions/setup-python@v4
        with:
          python-version: 3.11.2
      
      - name: checkout repo
        uses: actions/checkout@v3

      - name: install dbt adapters
        run: |
          pip install dbt-athena-community==1.4.1
          dbt deps
          dbt --version
      
      - name: sqlfluff
        uses: yu-iskw/action-sqlfluff@v3.6.0
        env:
            AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
            AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
            AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }} 
        id: lint-sql
        with:
          github_token: ${{ secrets.github_token }}
          reporter: github-pr-review
          sqlfluff_version: "1.4.5"
          sqlfluff_command: "fix" # Or "lint"
          config: "${{ github.workspace }}/.sqlfluff"
          paths: '${{ github.workspace }}/models'
          
      - name: 'Show outputs (Optional)'
        shell: bash
        run: |
          echo '${{ steps.lint-sql.outputs.sqlfluff-results }}' | jq -r '.'
          echo '${{ steps.lint-sql.outputs.sqlfluff-results-rdjson }}' | jq -r '.'

解决方案 #

1. 让SQLFluff环境加载dbt-athena适配器 #

yu-iskw/action-sqlfluff会创建独立Python虚拟环境，导致之前安装的dbt-athena-community无法被SQLFluff的dbt模板器调用。需在SQLFluff步骤中直接指定安装适配器：

- name: sqlfluff
  uses: yu-iskw/action-sqlfluff@v3.6.0
  env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }} 
  id: lint-sql
  with:
    github_token: ${{ secrets.github_token }}
    reporter: github-pr-review
    sqlfluff_version: "1.4.5"
    sqlfluff_command: "fix"
    config: "${{ github.workspace }}/.sqlfluff"
    paths: '${{ github.workspace }}/models'
    extra_packages: "dbt-athena-community==1.4.1" # 添加此行，确保适配器在SQLFluff环境中可用

2. 修复profiles.yml的凭证注入逻辑 #

GitHub Actions无本地AWS CLI配置的profile，需直接通过环境变量注入凭证：

default:
  outputs:
    dbt_dev:
      database: awsdatacatalog
      region_name: eu-west-2
      s3_staging_dir: s3://athena-query-results-XXX/
      schema: dbt_test
      type: athena
      work_group: primary
      # 从环境变量读取凭证
      aws_access_key_id: "{{ env_var('AWS_ACCESS_KEY_ID') }}"
      aws_secret_access_key: "{{ env_var('AWS_SECRET_ACCESS_KEY') }}"
  target: dbt_dev

3. 恢复.sqlfluff的dbt配置 #

之前移除的profile和target是SQLFluff识别dbt配置的关键参数，需恢复：

[sqlfluff:templater:dbt]
project_dir = ./
profiles_dir = ./
profile = default
target = dbt_dev

4. 验证适配器加载状态 #

在安装步骤后添加验证命令，确保dbt已正确加载athena适配器：

dbt --version
# 预期输出需包含：Plugins: dbt-athena-community: 1.4.1

内容的提问来源于stack exchange，提问作者mares