配置Stanford CoreNLP时遭遇GC overhead limit exceeded错误求助

问题现象 #

运行Stanford CoreNLP时触发以下内存异常：

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.lang.StringBuilder.toString(StringBuilder.java:407)
    at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3388)
    at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3183)
    at java.io.ObjectInputStream.readString(ObjectInputStream.java:1863)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1526)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
    at java.util.HashMap.readObject(HashMap.java:1402)
    at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2136)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
    at edu.stanford.nlp.io.IOUtils.readObjectFromURLOrClasspathOrFileSystem(IOUtils.java:310)
    at edu.stanford.nlp.coref.statistical.FeatureExtractor.loadVocabulary(FeatureExtractor.java:90)
    at edu.stanford.nlp.coref.statistical.FeatureExtractor.<init>(FeatureExtractor.java:75)
    at edu.stanford.nlp.coref.statistical.StatisticalCorefAlgorithm.<init>(StatisticalCorefAlgorithm.java:63)
    at edu.stanford.nlp.coref.statistical.StatisticalCorefAlgorithm.<init>(StatisticalCorefAlgorithm.java:44)
    at edu.stanford.nlp.coref.CorefAlgorithm.fromProps(CorefAlgorithm.java:30)
    at edu.stanford.nlp.coref.CorefSystem.<init>(CorefSystem.java:40)
    at edu.stanford.nlp.pipeline.CorefAnnotator.<init>(CorefAnnotator.java:69)
    at edu.stanford.nlp.pipeline.AnnotatorImplementations.coref(AnnotatorImplementations.java:218)
    at edu.stanford.nlp.pipeline.StanfordCoreNLP.lambda$getNamedAnnotators$17(StanfordCoreNLP.java:641)
    at edu.stanford.nlp.pipeline.StanfordCoreNLP$$Lambda$27/1579572132.apply(Unknown Source)
    at edu.stanford.nlp.pipeline.StanfordCoreNLP.lambda$null$33(StanfordCoreNLP.java:711)
    at edu.stanford.nlp.pipeline.StanfordCoreNLP$$Lambda$40/2104457164.get(Unknown Source)

系统环境 #

MacOS
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

触发命令 #

java edu.stanford.nlp.pipeline.StanfordCoreNLP -file input.txt

已确认配置 #

CLASSPATH设置：

:/Users/krishna/Downloads/NLP/stanford-corenlp-4.5.1/*

解决方案 #

• 增加JVM堆内存：Stanford CoreNLP的指代消解模块加载模型时需要大量内存，修改启动命令添加-Xmx参数分配更大堆内存：

  java -Xmx4g edu.stanford.nlp.pipeline.StanfordCoreNLP -file input.txt
  

  可根据你的Mac内存调整数值，比如8GB内存可设为-Xmx6g，16GB内存可设为-Xmx10g。

• 禁用不必要的Annotator：如果不需要指代消解（coref）功能，可指定仅加载所需模块，减少内存占用：

  java -Xmx2g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos -file input.txt
  

  按需添加其他Annotator，如lemma, ner等，避免加载默认包含的高内存模块。

• 升级Java版本：当前使用的Java 8u131版本较旧，升级到Java 8的最新补丁版或Java 11/17，新版本JVM在垃圾回收和内存管理上有优化，能缓解内存压力。

• 拆分大文件处理：若input.txt是超大文本，拆分为多个小文件分批处理，避免一次性加载过多数据占用内存。

内容的提问来源于stack exchange，提问作者kkgarg