AWS EMR中Spark Scala任务随机抛出org.xml.sax.SAXParseException异常求助

云原生阿岚

2026-7-24

问题描述

在AWS EMR（emr-5.31.0）上运行Spark(2.4.6) Scala任务时，随机出现org.xml.sax.SAXParseException; Premature end of file错误导致任务失败。该任务持续覆盖S3中的Parquet文件，多数情况下执行成功，但偶尔触发此错误。Spark UI无异常显示，无法定位问题根源。

堆栈跟踪

2023-04-19 01:04:50 ERROR FileFormatWriter:91 - Aborting job.
java.io.IOException: Failed publishing one or more staging directories
    at com.amazon.ws.emr.hadoop.fs.staging.ExternalStagedFileCommitter.lambda$publishOrDelete$0(ExternalStagedFileCommitter.java:69)
    at com.amazon.ws.emr.hadoop.fs.util.ExceptionCollector.throwIfNotEmpty(ExceptionCollector.java:89)
    at com.amazon.ws.emr.hadoop.fs.staging.ExternalStagedFileCommitter.publishOrDelete(ExternalStagedFileCommitter.java:68)
    at com.amazon.ws.emr.hadoop.fs.staging.DefaultStagingMechanism.publishOrDeleteExternalStagingDirectories(DefaultStagingMechanism.java:106)
    at org.apache.spark.internal.io.StagingServiceOptimizedCommitProtocol.commitJob(StagingServiceOptimizedCommitProtocol.scala:172)
    at org.apache.spark.internal.io.CompositeCommitProtocol.commitJob(CompositeCommitProtocol.scala:108)
    at org.apache.spark.sql.execution.datasources.SQLEmrOptimizedCommitProtocol.commitJob(SQLEmrOptimizedCommitProtocol.scala:121)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:187)
    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:173)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
    at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:173)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:169)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:197)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:194)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:169)
    at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:114)
    at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:112)
    at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:677)
    at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:677)
    at org.apache.spark.sql.execution.SQLExecution$.org$apache$spark$sql$execution$SQLExecution$$executeQuery$1(SQLExecution.scala:83)
    at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1$$anonfun$apply$1.apply(SQLExecution.scala:94)
    at org.apache.spark.sql.execution.QueryExecutionMetrics$.withMetrics(QueryExecutionMetrics.scala:141)
    at org.apache.spark.sql.execution.SQLExecution$.org$apache$spark$sql$execution$SQLExecution$$withMetrics(SQLExecution.scala:178)
    at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:93)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:200)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:92)
    at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:677)
    at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:286)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:272)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:230)
    at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:567)
    at ...write.parquet...
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:685)
Caused by: com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$DeleteObjectsHandler
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:166)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseDeletedObjectsResult(XmlResponsesSaxParser.java:472)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.transform.Unmarshallers$DeleteObjectsResultUnmarshaller.unmarshall(Unmarshallers.java:340)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.transform.Unmarshallers$DeleteObjectsResultUnmarshaller.unmarshall(Unmarshallers.java:336)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:44)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:30)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:69)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1726)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleSuccessResponse(AmazonHttpClient.java:1446)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1368)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5140)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5086)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2277)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.call.DeleteObjectsCall.perform(DeleteObjectsCall.java:24)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.call.DeleteObjectsCall.perform(DeleteObjectsCall.java:10)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor.execute(GlobalS3Executor.java:114)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:191)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:186)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.deleteObjects(AmazonS3LiteClient.java:128)
    at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.deleteAll(Jets3tNativeFileSystemStore.java:350)
    at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.doSingleThreadedBatchDelete(S3NativeFileSystem.java:1042)
    at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.delete(S3NativeFileSystem.java:343)
    at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.lambda$newFileCreationSubsystem$2(S3NativeFileSystem.java:217)
    at com.amazon.ws.emr.hadoop.fs.staging.ExternalStagedFileCommitter.deleteDir(ExternalStagedFileCommitter.java:341)
    at com.amazon.ws.emr.hadoop.fs.staging.ExternalStagedFileCommitter.access$000(ExternalStagedFileCommitter.java:38)
    at com.amazon.ws.emr.hadoop.fs.staging.ExternalStagedFileCommitter$1.lambda$newDirectoryTask$2(ExternalStagedFileCommitter.java:290)
    at com.amazon.ws.emr.hadoop.fs.staging.Task$1.run(Task.java:37)
    at com.amazon.ws.emr.hadoop.fs.staging.ExternalStagingTaskCoordinator$TaskWithCallback.run(ExternalStagingTaskCoordinator.java:152)
    at com.amazon.ws.emr.hadoop.fs.staging.StagedFilesExecutor.lambda$submitOrRun$0(StagedFilesExecutor.java:96)
    at com.amazon.ws.emr.hadoop.fs.staging.StagedFilesExecutor.submitOrRun(StagedFilesExecutor.java:104)
    at com.amazon.ws.emr.hadoop.fs.staging.StagedFilesExecutor.submitOrRunFirstBatch(StagedFilesExecutor.java:82)
    at com.amazon.ws.emr.hadoop.fs.staging.StagedFilesExecutor.run(StagedFilesExecutor.java:54)
    at com.amazon.ws.emr.hadoop.fs.staging.ExternalStagedFileCommitter.parallelExecute(ExternalStagedFileCommitter.java:266)
    at com.amazon.ws.emr.hadoop.fs.staging.ExternalStagedFileCommitter.parallelPublishOrDelete(ExternalStagedFileCommitter.java:247)
    at com.amazon.ws.emr.hadoop.fs.staging.ExternalStagedFileCommitter.publishOrDelete(ExternalStagedFileCommitter.java:63)
    ... 40 more
Caused by: org.xml.sax.SAXParseException; Premature end of file.
    at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
    at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
    at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)
    at org.apache.xerces.impl.XMLDocumentScannerImpl$PrologDispatcher.dispatch(Unknown Source)
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
    at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
    at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:152)
    ... 83 more

问题分析与解决方案

从堆栈跟踪可以看出，错误根源是Spark任务在提交阶段调用S3批量删除接口时，S3返回的XML响应被截断，导致解析失败。以下是针对性的解决办法：

增加任务重试次数：由于错误是随机出现的，提升任务重试阈值可以大概率规避。修改Spark配置：
```
spark.task.maxFailures=8
```
或者在代码中捕获写入异常，手动重试Parquet写入操作。
升级EMR版本：emr-5.31.0属于较老版本，后续的EMR 5.x版本（如emr-5.36.0）修复了S3客户端处理响应的多个bug。如果业务允许，也可以直接迁移到EMR 6.x系列，搭配Spark 3.x获得更稳定的S3交互能力。

切换文件提交器：禁用EMR优化的提交器，改用Spark原生的Hadoop提交器，避免EMR特定的提交逻辑触发异常。添加以下配置：

spark.sql.sources.commitProtocolClass=org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2

避免并发写入冲突：确保同一S3路径没有其他任务同时执行写入或删除操作。可以通过时间戳分区、分布式锁等方式隔离不同任务的写入目标，减少并发冲突概率。
检查S3存储类：如果目标路径使用了S3智能分层、Glacier等非标准存储类，可能存在访问延迟导致响应不完整。建议将写入路径切换为S3标准存储类。

内容的提问来源于stack exchange，提问作者sgallagher

AWS EMR中Spark Scala任务随机抛出org.xml.sax.SAXParseException异常求助

堆栈跟踪 #

堆栈跟踪