Delta表VACUUM LITE模式执行失败求助及相关疑问

背景 #

我们此前使用常规的VACUUM xxx RETAIN nnn HOURS语句清理Delta表，运行正常，但在大型数据库上执行耗时数小时。尝试使用新的VACUUM xxx LITE模式时，每次执行都会报错，报错信息如下：

org.apache.spark.sql.delta.DeltaIllegalStateException: [DELTA_CANNOT_VACUUM_LITE] VACUUM LITE cannot delete all eligible files as some files are not referenced by the Delta log. Please run VACUUM FULL.
    at org.apache.spark.sql.delta.DeltaErrorsBase.deltaCannotVacuumLite(DeltaErrors.scala:1724)
    at org.apache.spark.sql.delta.DeltaErrorsBase.deltaCannotVacuumLite$(DeltaErrors.scala:1723)
    at org.apache.spark.sql.delta.DeltaErrors$.deltaCannotVacuumLite(DeltaErrors.scala:3598)
    at org.apache.spark.sql.delta.commands.VacuumCommand$.getFilesFromDeltaLog(VacuumCommand.scala:530)
    at org.apache.spark.sql.delta.commands.VacuumCommand$.$anonfun$gc$1(VacuumCommand.scala:306)
    at org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:171)
    at org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:169)
    at org.apache.spark.sql.delta.commands.VacuumCommand$.recordFrameProfile(VacuumCommand.scala:58)
    at org.apache.spark.sql.delta.metering.DeltaLogging.$anonfun$recordDeltaOperationInternal$1(DeltaLogging.scala:139)
    at com.databricks.spark.util.DatabricksLogging.recordOperation(DatabricksLogging.scala:128)
    at com.databricks.spark.util.DatabricksLogging.recordOperation$(DatabricksLogging.scala:117)
    at org.apache.spark.sql.delta.commands.VacuumCommand$.recordOperation(VacuumCommand.scala:58)
    at org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperationInternal(DeltaLogging.scala:138)
    at org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperation(DeltaLogging.scala:128)
    at org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperation$(DeltaLogging.scala:118)
    at org.apache.spark.sql.delta.commands.VacuumCommand$.recordDeltaOperation(VacuumCommand.scala:58)
    at org.apache.spark.sql.delta.commands.VacuumCommand$.gc(VacuumCommand.scala:230)
    at io.delta.tables.execution.VacuumTableCommand.run(VacuumTableCommand.scala:67)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:126)
    at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:108)
    at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:385)
    at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:158)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$10(SQLExecution.scala:221)
    at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:108)
    at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:385)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$9(SQLExecution.scala:221)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:406)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:220)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:901)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:83)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:74)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:123)
    at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:114)
    at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:521)
    at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:77)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:521)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:34)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:303)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:299)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:34)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:34)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:497)
    at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:114)
    at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:101)
    at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:99)
    at org.apache.spark.sql.Dataset.<init>(Dataset.scala:223)
    at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:103)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:901)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:99)
    at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:639)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:901)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:630)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:660)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:569)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
    at java.base/java.lang.Thread.run(Thread.java:840)

即使在成功执行VACUUM xxx FULL后立即运行LITE模式，仍会触发该报错。

问题 #

1. 这是否意味着曾使用RETAIN HOURS执行过清理的表无法使用LITE模式？
2. 是否有用户成功使用过LITE模式执行清理操作？

环境配置 #

• AWS EMR 7.11
• Spark 3.5.6
• Delta 版本：3.3.2（依据AWS EMR发布文档）

解答 #

问题1：不是的 #

RETAIN HOURS只是指定清理保留时间的参数，本身不会导致表无法使用LITE模式。报错的核心原因是表的存储目录中存在Delta日志未引用的文件——这类文件可能来自：

• 手动删除Delta日志记录的文件后残留的空文件/碎片
• 旧版本Delta Lake的遗留元数据不一致问题
• 异常写入操作（比如写入中断后未被日志追踪的临时文件）
• 执行VACUUM FULL时未彻底清理所有未被追踪的文件

问题2：是的，大量用户已成功使用LITE模式 #

LITE模式是Delta Lake 3.0+推出的轻量清理方案，它仅通过Delta日志识别可删除文件，无需扫描整个存储目录，执行速度远快于FULL模式，适合作为日常周期性清理的首选。但它要求表的元数据完全一致，所有数据文件都被Delta日志正确记录，一旦存在日志外的文件就会触发报错。

内容的提问来源于stack exchange，提问作者Alexander Pavlov