Hey there! I totally get where you're coming from—Glue's default behavior with schema changes can feel counterintuitive when you're relying on Avro's built-in schema evolution support. Let's break down practical solutions to fix this and keep your data catalog clean while leveraging Avro's strengths:

1. Adjust Glue Crawler's Schema Change Policy #

By default, Glue crawlers create a new table whenever they detect a schema change—but you can tweak this behavior to update your existing table instead. Here's how:

• When creating/editing your crawler, navigate to the Schema change policy section.
• For Update behavior, select Update the table in the data catalog instead of Create a new table.
• For Delete behavior, choose an option that fits your needs:
  • DEPRECATE_IN_DATABASE: Marks removed fields as deprecated in the table schema (keeps them in the catalog for reference)
  • DELETE_FROM_DATABASE: Removes the fields entirely from the table schema
• If you prefer using the CLI, you can set this policy with:

  aws glue create-crawler --name your-crawler-name --schema-change-policy '{"UpdateBehavior": "UPDATE_IN_DATABASE", "DeleteBehavior": "DEPRECATE_IN_DATABASE"}' ...
  

This way, your existing table will automatically reflect new Avro fields without spawning duplicate tables.

2. Use AWS Glue Schema Registry for Centralized Schema Management #

Since Avro relies heavily on schema definitions, using Glue's Schema Registry adds a layer of control and consistency. Here's how to set it up:

• Create a Schema Registry in the Glue console (under Schema registry in the left sidebar).
• Register your base Avro schema, and enable schema evolution for the registry (allowing backward/forward compatible changes).
• When configuring your crawler's data source, link it to this Schema Registry. The crawler will now use the registry's schema versions to update your existing table instead of creating new ones.
  This is especially useful if you have multiple producers writing Avro files with evolving schemas—it ensures all data aligns with a single source of truth for schemas.

3. Build a Custom Glue ETL Job for Fine-Grained Control #

If the crawler's policy doesn't cover your specific use case, a custom Spark ETL job gives you full control over schema merging:

• Use Spark's native Avro support with the mergeSchema option to read all Avro files (old and new) with a merged schema:

  from awsglue.context import GlueContext
  from pyspark.context import SparkContext
  
  sc = SparkContext()
  glueContext = GlueContext(sc)
  
  # Read Avro files with merged schema
  df = glueContext.read.format("avro") \
      .option("mergeSchema", "true") \
      .load("s3://your-bucket/avro-data-path")
  
  # Write the merged data back to your Glue table
  glueContext.write_dynamic_frame.from_catalog(
      frame=glueContext.create_dynamic_frame.from_df(df, glueContext, "merged_df"),
      database="your-db",
      table_name="your-target-table",
      additional_options={"updateBehavior": "UPDATE_IN_DATABASE"}
  )
  

• You can schedule this job to run periodically, or trigger it when new Avro files are added to S3, ensuring your table schema stays in sync.

4. Clean Up Existing Duplicate Tables (If You Already Have Them) #

If you've already got multiple tables from past schema changes, you can merge them into a single master table:

• Use the merged schema approach from the custom ETL job above to read data from all existing tables.
• Write the combined data to a new master table (or overwrite the most recent table).
• Once verified, delete the old duplicate tables to keep your data catalog organized.

Key Notes to Remember #

• Always ensure your Avro schema changes are backward compatible (e.g., add optional fields, avoid removing required fields, don't change existing field data types). This prevents errors when reading old data with new schemas.
• After updating your crawler policy or running an ETL job, double-check the table schema in the Glue Data Catalog to confirm it matches your latest Avro files.
• Test any schema change strategy on a small subset of data first to avoid disrupting your production pipelines.

内容的提问来源于stack exchange，提问作者CharStar