使用Sedona与PySpark读取HDFS中Shapefile报错及解决记录

已验证的解决方案 #

• 传入Shapefile所在文件夹路径而非单个.shp文件路径给ShapefileReader.readToGeometryRDD方法
• 使用sedona-spark-shaded-3.0_2.12-1.4.0.jar包（该包整合了所有必要依赖）替代原分散的jar包引用

感谢Apache Sedona的Jia Yu提供的解决思路。

问题详情 #

原执行命令 #

spark-submit --jars /usr/local/spark/jars/sedona-core-3.0_2.13-1.4.0.jar,/usr/local/spark/jars/sedona-python-adapter-3.0_2.12-1.4.0.jar,/usr/local/spark/jars/sedona-sql-3.0_2.12-1.4.0.jar,/usr/local/spark/jars/geotools-wrapper-1.4.0-28.2.jar,/usr/local/spark/jars/sedona-python-adapter-3.0_2.12-1.4.0.jar pyspark_read_sedona.py

原Python代码 #

from pyspark.sql import SparkSession
from sedona.utils.adapter import Adapter
from sedona.register import SedonaRegistrator
from sedona.utils import SedonaKryoRegistrator, KryoSerializer

spark = SparkSession. \
    builder. \
    appName("NaturalEarthCities"). \
    config("spark.serializer", KryoSerializer.getName). \
    config("spark.kryo.registrator", SedonaKryoRegistrator.getName). \
    getOrCreate()

SedonaRegistrator.registerAll(spark)

from sedona.core.formatMapper.shapefileParser import ShapefileReader

# 原错误路径：指向单个.shp文件
shapefile_location = "hdfs:/naturalearth_lowres/naturalearth_lowres.shp"

spatial_rdd = ShapefileReader.readToGeometryRDD(spark.sparkContext, shapefile_location)

spatial_df = Adapter.toDf(spatial_rdd, spark)

spatial_df.createOrReplaceTempView("naturalearth_cities")

result_df = spark.sql("SELECT * FROM naturalearth_cities")
result_df.show()

报错信息 #

Traceback (most recent call last):
  File "/home/bigdata/ronnit/pyspark_read_sedona.py", line 21, in <module>
    spatial_rdd = ShapefileReader.readToGeometryRDD(spark.sparkContext, shapefile_location)
  File "/home/bigdata/anaconda3/lib/python3.7/site-packages/sedona/core/formatMapper/shapefileParser/shape_file_reader.py", line 42, in readToGeometryRDD
    inputPath
  File "/usr/local/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1322, in __call__
  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 190, in deco
  File "/usr/local/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.sedona.core.formatMapper.shapefileParser.ShapefileReader.readToGeometryRDD.
: java.lang.ArrayIndexOutOfBoundsException: 0
    at scala.collection.mutable.WrappedArray$ofRef.apply(WrappedArray.scala:193)
    at scala.collection.convert.Wrappers$SeqWrapper.get(Wrappers.scala:74)
    at org.apache.sedona.core.formatMapper.shapefileParser.ShapefileReader.readFieldNames(ShapefileReader.java:188)
    at org.apache.sedona.core.formatMapper.shapefileParser.ShapefileReader.readToGeometryRDD(ShapefileReader.java:82)
    at org.apache.sedona.core.formatMapper.shapefileParser.ShapefileReader.readToGeometryRDD(ShapefileReader.java:66)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
    at java.lang.Thread.run(Thread.java:745)

环境配置 #

Apache Sedona

Name: apache-sedona
Version: 1.4.0
Summary: Apache Sedona is a cluster computing system for processing large-scale spatial data
Home-page: https://sedona.apache.org
Author: Apache Sedona
Author-email: dev@sedona.apache.org
License: Apache License v2.0
Location: /home/bigdata/anaconda3/lib/python3.7/site-packages
Requires: shapely, attrs

PySpark

Name: pyspark
Version: 3.3.0
Summary: Apache Spark Python API
Home-page: https://github.com/apache/spark/tree/master/python
Author: Spark Developers
Author-email: dev@spark.apache.org
License: http://www.apache.org/licenses/LICENSE-2.0
Location: /home/bigdata/anaconda3/lib/python3.7/site-packages
Requires: py4j
Required-by: geospark

已完成的排查步骤 #

• 确认HDFS文件路径正确
• 本地通过geopandas可正常读取该Shapefile
• 验证Sedona和PySpark依赖已正确安装
• 确认文件权限为rw---r-r

内容的提问来源于stack exchange，提问作者Tinnor Retep