Spark读取含冒号(:)文件名的Parquet文件失败求助

问题场景 #

在Java环境下使用Spark读取路径为/home/admin/Parquet/DumpData/LC0:TCProfile.parquet的Parquet文件时，抛出如下异常：

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0) (executor driver): org.apache.spark.SparkException: Exception thrown in awaitResult: 
    at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)
    at org.apache.spark.util.ThreadUtils$.parmap(ThreadUtils.scala:375)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$.readParquetFootersInParallel(ParquetFileFormat.scala:508)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$.$anonfun$mergeSchemasInParallel$1(ParquetFileFormat.scala:556)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$.$anonfun$mergeSchemasInParallel$1$adapted(ParquetFileFormat.scala:549)
    at org.apache.spark.sql.execution.datasources.SchemaMergeUtils$.$anonfun$mergeSchemasInParallel$2(SchemaMergeUtils.scala:76)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:136)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Could not read footer for file: FileStatus{path=file:/home/admin/Parquet/DumpData/LC0:TCProfile.parquet; isDirectory=false; length=42694; replication=0; blocksize=0; modification_time=0; access_time=0; owner=; group=; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}
    at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotReadFooterForFileError(QueryExecutionErrors.scala:882)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$.$anonfun$readParquetFootersInParallel$1(ParquetFileFormat.scala:521)
    at org.apache.spark.util.ThreadUtils$.$anonfun$parmap$2(ThreadUtils.scala:372)
    at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
    at scala.util.Success.$anonfun$map$1(Try.scala:255)
    at scala.util.Success.map(Try.scala:213)
    at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
    at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
    at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
    at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: .LC0:TCProfile.parquet.crc
    at org.apache.hadoop.fs.Path.initialize(Path.java:263)
    at org.apache.hadoop.fs.Path.<init>(Path.java:221)
    at org.apache.hadoop.fs.Path.<init>(Path.java:129)
    at org.apache.hadoop.fs.ChecksumFileSystem.getChecksumFile(ChecksumFileSystem.java:107)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:163)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:372)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:976)
    at org.apache.parquet.hadoop.util.HadoopInputFile.newStream(HadoopInputFile.java:69)

由于无法修改文件名（无重命名权限），需通过其他方式解决。

异常根源 #

核心错误是java.net.URISyntaxException: Relative path in absolute URI: .LC0:TCProfile.parquet.crc，Hadoop的ChecksumFileSystem在生成CRC校验文件路径时，将文件名中的冒号误判为URI的scheme分隔符，导致路径解析失败。

解决方案 #

方案1：禁用Hadoop校验和检查 #

将Hadoop的文件系统实现替换为不处理校验和的LocalFileSystem，避免生成和读取CRC文件：

• 在Spark代码中配置：

SparkConf conf = new SparkConf()
    .set("spark.hadoop.fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem");
SparkSession spark = SparkSession.builder().config(conf).getOrCreate();

• 或在提交Spark作业时通过命令行参数设置：

spark-submit --conf spark.hadoop.fs.file.impl=org.apache.hadoop.fs.LocalFileSystem your-app.jar

方案2：使用RawLocalFileSystem #

RawLocalFileSystem是LocalFileSystem的子类，同样跳过校验和处理，配置方式类似：

• 代码中配置：

SparkConf conf = new SparkConf()
    .set("spark.hadoop.fs.file.impl", "org.apache.hadoop.fs.RawLocalFileSystem");
SparkSession spark = SparkSession.builder().config(conf).getOrCreate();

• 命令行参数：

spark-submit --conf spark.hadoop.fs.file.impl=org.apache.hadoop.fs.RawLocalFileSystem your-app.jar

方案3：关闭Schema合并 #

异常发生在Spark并行读取Parquet Footer进行Schema合并的阶段。如果已知目标Parquet文件的Schema固定，可直接关闭Schema合并，跳过Footer读取操作：

Dataset<Row> df = spark.read()
    .option("mergeSchema", "false")
    .parquet("/home/admin/Parquet/DumpData/LC0:TCProfile.parquet");

方案选择 #

• 方案1、2：适用于所有本地文件读取场景，彻底规避CRC文件的URI解析问题；
• 方案3：仅适用于Schema固定的单文件读取场景，无需修改Hadoop底层配置。

内容的提问来源于stack exchange，提问作者Ram Shan