如何通过CloudFormation在Glue Crawler中指定Glue表与Schema实现自动更新

你需要对原有模板做4处核心修改，即可实现Crawler自动同步myTestTable的字段新增、删除变更：

• 移除GlueCrawler的TablePrefix配置：该配置会让Crawler生成带指定前缀的新表，无法匹配你预先创建的myTestTable
• 调整SchemaChangePolicy删除行为：原有DeleteBehavior: LOG仅会记录字段删除操作，不会同步到表结构，改为DELETE_FROM_DATABASE即可实现字段删除的自动同步
• 补充Crawler Configuration配置：指定合并已有表Schema、禁止生成多余新表，确保Crawler只会更新匹配到的myTestTable
• 为GlueTable资源添加保留策略：避免后续CloudFormation栈更新时，用模板中固定的初始Schema覆盖Crawler同步后的最新表结构

修改后的完整模板片段如下：

GlueDatabase:
  Type: AWS::Glue::Database
  Properties:
    CatalogId: !Ref AWS::AccountId
    DatabaseInput:
      Name: myTestGlueDB

GlueTable:
  Type: AWS::Glue::Table
  DeletionPolicy: Retain
  UpdateReplacePolicy: Retain
  Properties:
    CatalogId: !Ref AWS::AccountId
    DatabaseName: !Ref GlueDatabase
    TableInput:
      Parameters:
        classification: parquet
      Name: myTestTable
      Owner: owner
      Retention: 0
      StorageDescriptor:
        Columns:
        - Name: productName
          Type: string
        - Name: productId
          Type: string
        InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
        OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
        Compressed: false
        NumberOfBuckets: -1
        SerdeInfo:
          SerializationLibrary: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
          Parameters:
            serialization.format: '1'
        Location: s3://test-bucket
        BucketColumns: []
        SortColumns: []
        StoredAsSubDirectories: false
      PartitionKeys:
      - Name: id
        Type: string
      - Name: year
        Type: string
      - Name: month
        Type: string
      - Name: day
        Type: string
      TableType: EXTERNAL_TABLE

GlueCrawler:
  Type: AWS::Glue::Crawler
  Properties:
    Name: myTestCrawler
    Role: GlueCrawlerRole
    DatabaseName: myTestGlueDB
    Targets:
      S3Targets:
        - Path: s3://test-bucket
    SchemaChangePolicy:
      UpdateBehavior: UPDATE_IN_DATABASE
      DeleteBehavior: DELETE_FROM_DATABASE
    Configuration: |
      {
        "Version": 1.0,
        "Grouping": {
          "TableGroupingPolicy": "CombineCompatibleSchemas"
        },
        "CrawlerOutput": {
          "Tables": {
            "AddOrUpdateBehavior": "MergeExistingSchemas"
          }
        },
        "CreateTablesOnlyIfNoMatchingTables": false
      }

配置生效后，Crawler运行时会自动匹配S3路径对应的已有myTestTable，将数据源的Schema变更同步到表结构中。

内容的提问来源于stack exchange，提问作者Mike Marsh