Spark从S3加载图片触发NumberFormatException错误求助

重现代码 #

# Chargement des packages et options de configuration

import os
os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages com.amazonaws:aws-java-sdk-pom:1.10.34,org.apache.hadoop:hadoop-aws:2.7.3,databricks:spark-deep-learning:1.5.0-spark2.4-s_2.11 pyspark-shell --conf spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true",spark.hadoop.fs.s3a.endpoint=s3.eu-west-1.amazonaws.com'
# Creating SparkSession
sc = SparkContext.getOrCreate()
spark = SparkSession.builder \
    .appName('Image_P8') \
    .config('spark.driver.extraJavaOptions', '-Dio.netty.tryReflectionSetAccessible=true') \
    .config('spark.hadoop.fs.s3a.endpoint', 's3.eu-west-1.amazonaws.com') \
    .getOrCreate()
path = "s3a://ocr-fruits/Test/*"
image_df = spark.read.format("binaryFile") \
  .option("pathGlobFilter", "*.jpg") \
  .option("recursiveFileLookup", "true") \
  .load(path)

image_df.show(2)

错误栈 #

NumberFormatException                     Traceback (most recent call last)
Cell In[65], line 4
      1 image_df = spark.read.format("binaryFile") \
      2   .option("pathGlobFilter", "*.jpg") \
      3   .option("recursiveFileLookup", "true") \
----> 4   .load(path)
      6 image_df.show(2)

File C:\apps\opt\spark-3.4.0-bin-hadoop3\python\pyspark\sql\readwriter.py:300, in DataFrameReader.load(self, path, format, schema, **options)
    298 self.options(**options)
    299 if isinstance(path, str):
--> 300     return self._df(self._jreader.load(path))
    301 elif path is not None:
    302     if type(path) != list:

File C:\apps\opt\spark-3.4.0-bin-hadoop3\python\lib\py4j-0.10.9.7-src.zip\py4j\java_gateway.py:1322, in JavaMember.__call__(self, *args)
   1316 command = proto.CALL_COMMAND_NAME +\
   1317     self.command_header +\
   1318     args_command +\
   1319     proto.END_COMMAND_PART
   1321 answer = self.gateway_client.send_command(command)
--> 1322 return_value = get_return_value(
   1323     answer, self.gateway_client, self.target_id, self.name)
   1325 for temp_arg in temp_args:
   1326     if hasattr(temp_arg, "_detach"):

File C:\apps\opt\spark-3.4.0-bin-hadoop3\python\pyspark\errors\exceptions\captured.py:175, in capture_sql_exception.<locals>.deco(*a, **kw)
    171 converted = convert_exception(e.java_exception)
    172 if not isinstance(converted, UnknownException):
    173     # Hide where the exception came from that shows a non-Pythonic
    174     # JVM exception message.
--> 175     raise converted from None
    176 else:
    177     raise

NumberFormatException: For input string: "64M"

原因分析 #

核心问题是Hadoop AWS客户端版本与Spark版本不兼容：

• 你使用的Spark为3.4.0（基于Hadoop 3.x），但搭配了旧版本的hadoop-aws:2.7.3（属于Hadoop 2.x系列）。
• 旧版Hadoop AWS客户端处理内存配置参数时，无法识别带单位（如M）的数值，会直接尝试将64M转换为纯数字，从而触发NumberFormatException。

解决方案 #

1. 替换兼容的依赖包 #

Spark 3.4.0对应Hadoop版本为3.3.4，需改用匹配版本的AWS相关依赖：

• 替换org.apache.hadoop:hadoop-aws:2.7.3为org.apache.hadoop:hadoop-aws:3.3.4
• 替换com.amazonaws:aws-java-sdk-pom:1.10.34为com.amazonaws:aws-java-sdk-bundle:1.12.590（该版本兼容Hadoop 3.x）
• 若使用databricks:spark-deep-learning，需确保版本适配Spark 3.x，或直接使用Spark原生binaryFile格式完成图片加载。

2. 修正PYSPARK_SUBMIT_ARGS配置 #

原配置中多个--conf参数用逗号分隔错误，需改为空格分隔；同时更新依赖包：

os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages com.amazonaws:aws-java-sdk-bundle:1.12.590,org.apache.hadoop:hadoop-aws:3.3.4 pyspark-shell --conf spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" --conf spark.hadoop.fs.s3a.endpoint=s3.eu-west-1.amazonaws.com'

3. 调整SparkSession配置 #

确保SparkSession配置与提交参数一致，或直接依赖提交参数避免重复配置：

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName('Image_P8') \
    .getOrCreate()

path = "s3a://ocr-fruits/Test/*"
image_df = spark.read.format("binaryFile") \
  .option("pathGlobFilter", "*.jpg") \
  .option("recursiveFileLookup", "true") \
  .load(path)

image_df.show(2)

额外注意事项 #

• 需配置AWS凭证：可通过~/.aws/credentials文件，或设置环境变量AWS_ACCESS_KEY_ID、AWS_SECRET_ACCESS_KEY。
• 若仍有访问问题，可在Spark配置中直接添加凭证参数：

  spark = SparkSession.builder \
      .appName('Image_P8') \
      .config("spark.hadoop.fs.s3a.access.key", "你的访问密钥") \
      .config("spark.hadoop.fs.s3a.secret.key", "你的秘密密钥") \
      .getOrCreate()
  

内容的提问来源于stack exchange，提问作者Bydhon