为何Delta Lake存储大量冗余数据？覆盖写入疑问

Great question—this is a super common misconception when you're new to Delta Lake, especially since the name makes it easy to draw a direct parallel to Git's delta-only commits. Let's break down what's happening here and why Delta behaves this way.

First, Clarifying Delta Lake's Storage Model #

Contrary to what the name might imply, Delta Lake isn't a "delta-only" storage system like Git. Instead, it's a versioned data lake that maintains full snapshots of your table at each commit, while using file-level deltas to optimize storage and access. Each commit adds new files (for inserts/updates) or marks existing files as deleted (for overwrites/deletes), but it doesn't inherently compare file contents to avoid duplicates during writes.

Why Your "Empty" Overwrite Creates Duplicate Files #

When you use .mode("overwrite") with Delta Lake, here's the play-by-play under the hood:

• Delta treats this as a full overwrite operation by default. It doesn't check if the incoming data matches the existing table—instead, it first writes the new data files to your storage location.
• Once the new files are successfully written, it updates the transaction log (_delta_log) to mark the old files as "deleted" from the current table version.
• Crucially, Delta doesn't auto-delete old files right away (they're retained for time travel and recovery, default 7 days), so you see both old and new files in your S3 bucket until you run a VACUUM command.

The reason Delta skips automatic content-based deduplication during overwrite is performance. In big data scenarios, comparing every incoming file to every existing file to check for duplicates would add massive overhead—reading, hashing, and validating terabytes of data would grind writes to a halt. Delta prioritizes write speed and ACID guarantees over implicit deduplication.

How to Avoid This Behavior #

If you want to skip creating duplicate files when writing identical data, here are a few practical approaches:

1. Use MERGE instead of OVERWRITE:
   Instead of overwriting the entire table, use Delta's MERGE operation to only update rows that have changed. If your incoming data is identical to the existing table, MERGE won't generate any new files.

   import io.delta.tables._
   
   val deltaTable = DeltaTable.forPath(spark, delta_base_path)
   deltaTable.as("target")
     .merge(df.as("source"), "target.id = source.id")
     .whenMatchedUpdateAll()
     .whenNotMatchedInsertAll()
     .execute()
   

2. Pre-check for data differences:
   Run a quick validation to see if the incoming DataFrame differs from the existing table before writing. For example:

   val existingDF = spark.read.format("delta").load(delta_base_path)
   if (!df.except(existingDF).isEmpty) {
     df.write.format("delta").mode("overwrite").save(delta_base_path)
   }
   

   Note: This works well for small-to-medium datasets, but might not be efficient for extremely large tables.
3. Clean up old files with VACUUM:
   Even if you do generate duplicate files, you can clean up unused old files to free storage using the VACUUM command:

   deltaTable.vacuum(0) // 0 means delete all files not needed for the current version (use cautiously!)
   

   By default, VACUUM retains files for 7 days to support time travel functionality.

Recap of Your Observed Behavior #

To tie this back to your example:

• First write: Creates initial data files (part-00000-UUID1, part-00001-UUID2) and transaction log 00000000000000000000.json.
• Second overwrite with identical data: Writes new files (part-00000-UUID3, part-00001-UUID4), adds transaction log 00000000000000000001.json marking the old files as deleted. The old files remain in storage until VACUUM is executed.

内容的提问来源于stack exchange，提问作者bensenberner