Geomesa 4.0.1导入Shapefile遇投影及列数超限报错求助

环境版本 #

• GeoMesa 4.0.1
• Hadoop 2.10.2
• Apache Accumulo 2.0.1
• Apache Zookeeper 3.7.1

导入命令 #

geomesa-accumulo ingest --force -i accumulo -z zookeeper -u username -p myPassword -c myCatalog -f myFeature  /path/to/shapefile

报错1：投影转换异常 #

2023-07-26 06:50:24,914 ERROR [org.locationtech.geomesa.tools.ingest.LocalConverterIngest] Fatal error running local ingest worker on /data/hdfs/file_geomesa/CD_Toscana/Gasdotto-45840-CD-Maggio-centroidi.shp
java.io.IOException: Error occurred trying to reproject data
        at org.geotools.data.store.ContentFeatureSource.getReader(ContentFeatureSource.java:723)
        at org.locationtech.geomesa.convert.shp.ShapefileConverter.parse(ShapefileConverter.scala:74)
        at org.locationtech.geomesa.convert2.AbstractConverter.process(AbstractConverter.scala:151)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$4(LocalConverterIngest.scala:172)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$4$adapted(LocalConverterIngest.scala:168)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:575)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:573)
        at org.locationtech.geomesa.utils.collection.CloseableIterator$CloseableSingleIterator.foreach(CloseableIterator.scala:85)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$3(LocalConverterIngest.scala:168)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$3$adapted(LocalConverterIngest.scala:167)
        at org.locationtech.geomesa.utils.io.package$WithClose$.apply(package.scala:64)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$2(LocalConverterIngest.scala:167)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$2$adapted(LocalConverterIngest.scala:166)
        at org.locationtech.geomesa.utils.io.CloseablePool$CommonsPoolPool.borrow(CloseablePool.scala:68)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.run(LocalConverterIngest.scala:166)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Nothing to be reprojected! (check before using wrapper)
        at org.geotools.data.crs.ReprojectFeatureReader.<init>(ReprojectFeatureReader.java:152)
        at org.geotools.data.crs.ReprojectFeatureReader.<init>(ReprojectFeatureReader.java:117)
        at org.geotools.data.store.ContentFeatureSource.getReader(ContentFeatureSource.java:719)
        ... 19 more

排查过程 #

• 最初猜测是要素数量过多导致，但测试少量要素的文件仍触发相同报错
• 该文件在GeoMesa 3.5.2及以下版本可正常导入，4.0.0及以上版本报错
• 删除Shapefile对应的.prj文件后，少量要素文件可正常导入，但列数超过600的文件仍报错

报错2：数组越界异常 #

at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter.writeFeature(GeoMesaFeatureWriter.scala:56)
        at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter.writeFeature$(GeoMesaFeatureWriter.scala:46)
        at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter$TableFeatureWriter.writeFeature(GeoMesaFeatureWriter.scala:151)
        at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter$GeoMesaAppendFeatureWriter.write(GeoMesaFeatureWriter.scala:239)
        at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter$GeoMesaAppendFeatureWriter.write$(GeoMesaFeatureWriter.scala:235)
        at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter$$anon$3.write(GeoMesaFeatureWriter.scala:111)
        at org.locationtech.geomesa.utils.geotools.FeatureUtils$.write(FeatureUtils.scala:147)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$8(LocalConverterIngest.scala:181)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$8$adapted(LocalConverterIngest.scala:179)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:575)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:573)
        at org.locationtech.geomesa.utils.collection.CloseableIterator$FlatMapCloseableIterator.foreach(CloseableIterator.scala:132)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$7(LocalConverterIngest.scala:179)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$7$adapted(LocalConverterIngest.scala:173)
        at org.locationtech.geomesa.utils.io.CloseablePool$CommonsPoolPool.borrow(CloseablePool.scala:68)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$6(LocalConverterIngest.scala:173)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$6$adapted(LocalConverterIngest.scala:172)
        at org.locationtech.geomesa.utils.io.package$WithClose$.apply(package.scala:64)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$4(LocalConverterIngest.scala:172)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$4$adapted(LocalConverterIngest.scala:168)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:575)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:573)
        at org.locationtech.geomesa.utils.collection.CloseableIterator$CloseableSingleIterator.foreach(CloseableIterator.scala:85)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$3(LocalConverterIngest.scala:168)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$3$adapted(LocalConverterIngest.scala:167)
        at org.locationtech.geomesa.utils.io.package$WithClose$.apply(package.scala:64)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$2(LocalConverterIngest.scala:167)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.$anonfun$run$2$adapted(LocalConverterIngest.scala:166)
        at org.locationtech.geomesa.utils.io.CloseablePool$CommonsPoolPool.borrow(CloseablePool.scala:68)
        at org.locationtech.geomesa.tools.ingest.LocalConverterIngest$LocalIngestWorker.run(LocalConverterIngest.scala:166)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 2136
        at com.esotericsoftware.kryo.io.Output.writeByte(Output.java:226)
        at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.locationtech.geomesa.features.serialization.WkbSerialization.serializeWkb(WkbSerialization.scala:44)
        at org.locationtech.geomesa.features.serialization.WkbSerialization.serializeWkb$(WkbSerialization.scala:42)
        at org.locationtech.geomesa.features.kryo.serialization.KryoGeometrySerialization$.serializeWkb(KryoGeometrySerialization.scala:14)
        at org.locationtech.geomesa.features.kryo.impl.KryoFeatureSerialization$KryoGeometryWkbWriter$.apply(KryoFeatureSerialization.scala:229)
        at org.locationtech.geomesa.features.kryo.impl.KryoFeatureSerialization.writeFeature(KryoFeatureSerialization.scala:71)
        at org.locationtech.geomesa.features.kryo.impl.KryoFeatureSerialization.serialize(KryoFeatureSerialization.scala:43)
        at org.locationtech.geomesa.features.kryo.impl.KryoFeatureSerialization.serialize$(KryoFeatureSerialization.scala:41)
        at org.locationtech.geomesa.features.kryo.KryoFeatureSerializer$MutableActiveSerializer.serialize(KryoFeatureSerializer.scala:75)
        at org.locationtech.geomesa.index.api.WritableFeature$FeatureLevelWritableFeature.$anonfun$values$2(WritableFeature.scala:153)
        at org.locationtech.geomesa.index.api.package$KeyValue.value$lzycompute(package.scala:183)
        at org.locationtech.geomesa.index.api.package$KeyValue.value(package.scala:183)
        at org.locationtech.geomesa.accumulo.data.AccumuloIndexAdapter$AccumuloIndexWriter.$anonfun$write$1(AccumuloIndexAdapter.scala:397)
        at org.locationtech.geomesa.accumulo.data.AccumuloIndexAdapter$AccumuloIndexWriter.$anonfun$write$1$adapted(AccumuloIndexAdapter.scala:396)
        at scala.collection.immutable.Vector.foreach(Vector.scala:1895)
        at org.locationtech.geomesa.accumulo.data.AccumuloIndexAdapter$AccumuloIndexWriter.write(AccumuloIndexAdapter.scala:396)
        at org.locationtech.geomesa.index.api.IndexAdapter$BaseIndexWriter.write(IndexAdapter.scala:153)
        at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter.writeFeature(GeoMesaFeatureWriter.scala:50)
        ... 34 more

疑问 #

请问该问题是GeoMesa新版本引入的bug，还是需要针对新版本调整配置？

内容的提问来源于stack exchange，提问作者Luigi