GraphDB预加载24GB Trig文件遇格式错误，求忽略错误方案

我尝试使用GraphDB的preload命令预加载一个约24GB的Trig备份文件，执行的Docker命令如下：

docker run -v $(pwd)/graphdb-data:/opt/graphdb/home \
       -v $(pwd)/preload:/opt/graphdb-import \
       --entrypoint /opt/graphdb/dist/bin/importrdf \
       -e GDB_JAVA_OPTS="-Xmx12g -Xms2g" \
       -e graphdb.page.cache.size=512m \
       -e graphdb.workers.limit=2 \
       -e graphdb.query.evaluation.mode=disk \
       -e graphdb.repository.index.enable=false \
       -e graphdb.compression.enabled=true \
       -e graphdb.use.native.jena.model=true -e graphdb.verify-literals=false \
       ontotext/graphdb:10.7.1 \
       preload -s --force --recursive -q /tmp -c /opt/graphdb-import/graphdb-repo.ttl /opt/graphdb-import/backup.trig

数据加载接近完成时，出现如下错误：

12:05:31.663 [resolver] INFO  c.ontotext.graphdb.importrdf.Preload - 310,000,000 statements ...
12:06:30.853 [resolver] INFO  c.ontotext.graphdb.importrdf.Preload - 320,000,000 statements ...
12:06:50.697 [monitor file position] INFO  c.ontotext.graphdb.importrdf.Preload - File backup.trig processed to position 23,774,363,648 from 26,664,959,488 bytes
12:07:54.327 [resolver] INFO  c.ontotext.graphdb.importrdf.Preload - 330,000,000 statements ...
12:08:50.702 [monitor file position] INFO  c.ontotext.graphdb.importrdf.Preload - File backup.trig processed to position 25,267,535,872 from 26,664,959,488 bytes
12:09:28.692 [resolver] INFO  c.ontotext.graphdb.importrdf.Preload - 340,000,000 statements ...
java.lang.NumberFormatException: empty String
    at java.base/jdk.internal.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1842)
    at java.base/jdk.internal.math.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
    at java.base/java.lang.Double.parseDouble(Double.java:543)
    at org.eclipse.rdf4j.model.base.AbstractLiteral$NumberLiteral.parseDouble(AbstractLiteral.java:367)
    at java.base/java.util.Optional.map(Optional.java:265)
    at org.eclipse.rdf4j.model.base.AbstractLiteral.value(AbstractLiteral.java:100)
    at org.eclipse.rdf4j.model.base.AbstractLiteral.doubleValue(AbstractLiteral.java:141)
    at com.ontotext.graphdb.importrdf.Preload.processLiteral(Preload.java:2124)
    at com.ontotext.graphdb.importrdf.Resolver.write(Resolver.java:61)
    at com.ontotext.graphdb.importrdf.Resolver.createid(Resolver.java:124)
    at com.ontotext.graphdb.importrdf.Resolver.run(Resolver.java:203)
java.lang.InterruptedException
    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2090)
    at java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
    at com.ontotext.graphdb.importrdf.Preload$LocalHandler.handleStatement(Preload.java:434)
    at org.eclipse.rdf4j.rio.trig.TriGParser.reportStatement(TriGParser.java:248)
    at org.eclipse.rdf4j.rio.turtle.TurtleParser.parseObject(TurtleParser.java:453)
    at org.eclipse.rdf4j.rio.turtle.TurtleParser.parseObjectList(TurtleParser.java:374)
    at org.eclipse.rdf4j.rio.turtle.TurtleParser.parsePredicateObjectList(TurtleParser.java:347)
    at org.eclipse.rdf4j.rio.trig.TriGParser.parseTriples(TriGParser.java:236)
    at org.eclipse.rdf4j.rio.trig.TriGParser.parseGraph(TriGParser.java:163)
    at org.eclipse.rdf4j.rio.trig.TriGParser.parseStatement(TriGParser.java:115)
    at org.eclipse.rdf4j.rio.turtle.TurtleParser.parse(TurtleParser.java:164)
    at org.eclipse.rdf4j.repository.util.RDFLoader.loadInputStreamOrReader(RDFLoader.java:304)
    at org.eclipse.rdf4j.repository.util.RDFLoader.load(RDFLoader.java:249)
    at com.ontotext.load.GraphdbRDFLoader.load(GraphdbRDFLoader.java:89)
    at com.ontotext.graphdb.importrdf.Preload.processSingleFileInternal(Preload.java:2103)
    at com.ontotext.graphdb.importrdf.Preload.lambda$processSingleFile$22(Preload.java:2053)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
12:10:32.001 [sorting] INFO  c.ontotext.graphdb.importrdf.Preload - Sorter thread finished.

该文件是开发环境实例的直接备份，我希望能忽略此类类型推断错误，仅导入正确数据，请问通过GraphDB的preload命令是否可以实现？

GraphDB的preload工具本身没有直接暴露忽略这类类型解析错误的命令行参数，但可以通过以下几种方式实现跳过错误继续导入：

1. 添加JVM参数跳过损坏字面量 #

在GDB_JAVA_OPTS中添加-Dgraphdb.importrdf.skip-bad-literals=true，这个参数会让preload工具跳过无法解析的字面量语句，继续处理其他正确数据。修改后的Docker命令如下：

docker run -v $(pwd)/graphdb-data:/opt/graphdb/home \
       -v $(pwd)/preload:/opt/graphdb-import \
       --entrypoint /opt/graphdb/dist/bin/importrdf \
       -e GDB_JAVA_OPTS="-Xmx12g -Xms2g -Dgraphdb.importrdf.skip-bad-literals=true" \
       -e graphdb.page.cache.size=512m \
       -e graphdb.workers.limit=2 \
       -e graphdb.query.evaluation.mode=disk \
       -e graphdb.repository.index.enable=false \
       -e graphdb.compression.enabled=true \
       -e graphdb.use.native.jena.model=true -e graphdb.verify-literals=false \
       ontotext/graphdb:10.7.1 \
       preload -s --force --recursive -q /tmp -c /opt/graphdb-import/graphdb-repo.ttl /opt/graphdb-import/backup.trig

2. 预处理Trig文件修复错误 #

如果上述参数无效，可以先定位并修复文件中的错误语句：

• 用GraphDB的validate工具扫描文件，找到错误位置：

docker run --rm -v $(pwd)/preload:/opt/graphdb-import ontotext/graphdb:10.7.1 /opt/graphdb/dist/bin/validate /opt/graphdb-import/backup.trig

• 用RDF4J的rio转换工具跳过错误语句，生成修复后的文件：

docker run --rm -v $(pwd)/preload:/opt/graphdb-import ontotext/graphdb:10.7.1 /opt/graphdb/dist/bin/rio convert --skip-bad-rdf -i trig -o trig /opt/graphdb-import/backup.trig /opt/graphdb-import/fixed-backup.trig

之后用修复后的fixed-backup.trig执行preload导入。

3. 切换到常规导入方式 #

如果preload工具的错误容忍度无法满足需求，可以尝试使用GraphDB工作台的导入功能或REST API进行导入，这两种方式默认具备更好的错误处理能力，可配置跳过错误语句继续导入。

内容的提问来源于stack exchange，提问作者IVI