使用airflow connections import时Database字段设置及异常排查

问题现象 #

• 执行airflow connections import命令时触发错误：对象包含非法键database，仅允许使用指定键集合：{'login', 'password', 'schema', 'description', 'conn_type', 'port', 'conn_id', 'host', 'extra_dejson', 'extra', 'uri'}
• Airflow Web界面中PostgreSQL连接明确存在Database输入字段，与命令行导入的字段要求不一致
• 导入完成后，连接类型被默认设置为Amazon Chime Webhook，而非预期的postgresql

用户提供的原始连接文件 #

{
    "landing_db": {
        "host": "data-pipeline-db",
        "database": "test-db",
        "description": "",
        "port": "5432",
        "login": "test",
        "password": "test",
        "conn_type": "postgresql"},
    "time_series": {
        "host": "data-pipeline-db",
        "description": "",
        "database": "test-db",
        "port": "5432",
        "login": "test",
        "password": "test",
        "conn_type": "postgresql"
    }
}

解决步骤 #

1. 替换字段名：将database改为schema
   Airflow的命令行导入接口与Web界面存在字段命名差异，PostgreSQL连接的数据库名称在API层面使用schema键，而非界面显示的database。修改后的连接文件如下：

   {
       "landing_db": {
           "host": "data-pipeline-db",
           "schema": "test-db",
           "description": "",
           "port": "5432",
           "login": "test",
           "password": "test",
           "conn_type": "postgresql"},
       "time_series": {
           "host": "data-pipeline-db",
           "description": "",
           "schema": "test-db",
           "port": "5432",
           "login": "test",
           "password": "test",
           "conn_type": "postgresql"
       }
   }
   

2. 确认conn_type字段正确性
   连接类型被默认设置为Amazon Chime Webhook是因为导入时conn_type识别失败。确保修改后的文件中每个连接的conn_type都明确设置为postgresql，Airflow会根据该字段自动匹配正确的连接类型。

3. 重新执行导入命令
   使用修改后的文件重新运行导入命令：

   airflow connections import your_connections_file.json
   

内容的提问来源于stack exchange，提问作者AlexLordThorsen