如何在Hibernate Search 5.6.1中使用WordNet同义词？

我之前在基于Hibernate Search 5.x开发搜索功能时，也碰到过用WordNet同义词替代静态文件的需求，刚好可以给你分享两种可行的方案：

方案一：预生成WordNet同义词静态文件 #

这种方式适合同义词不常更新的场景，先把WordNet里的同义词提取成Solr格式的静态文件，再交给SynonymFilterFactory使用。

步骤1：引入WordNet访问依赖 #

用JWNL（Java WordNet Library）来读取WordNet词库，在Maven中添加依赖：

<dependency>
    <groupId>net.sf.jwnl</groupId>
    <artifactId>jwnl</artifactId>
    <version>1.4.1</version>
</dependency>

步骤2：编写工具类提取同义词 #

写一个简单的Java程序，遍历WordNet的词库，提取同义词并输出成Solr兼容的格式（一行一个词项，同义词用逗号分隔）：

import net.sf.jwnl.JWNL;
import net.sf.jwnl.data.IndexWord;
import net.sf.jwnl.data.POS;
import net.sf.jwnl.data.Synset;
import net.sf.jwnl.data.Word;
import net.sf.jwnl.dictionary.Dictionary;

import java.io.FileWriter;
import java.io.IOException;
import java.util.HashSet;
import java.util.Set;

public class WordNetSynonymExporter {
    public static void main(String[] args) throws Exception {
        // 初始化JWNL（需要jwnl_properties.xml配置文件，指定WordNet路径）
        JWNL.initialize(WordNetSynonymExporter.class.getResourceAsStream("/jwnl_properties.xml"));
        Dictionary dictionary = Dictionary.getInstance();
        
        FileWriter writer = new FileWriter("src/main/resources/wordnet_synonyms.txt");
        
        // 遍历名词、动词、形容词等词性，按需调整
        for (POS pos : new POS[]{POS.NOUN, POS.VERB}) {
            IndexWord indexWord = dictionary.getIndexWordIterator(pos).next();
            while (indexWord != null) {
                Set<String> synonyms = new HashSet<>();
                // 添加当前词
                synonyms.add(indexWord.getLemma());
                // 获取同义词集
                for (Synset synset : indexWord.getSenses()) {
                    for (Word word : synset.getWords()) {
                        synonyms.add(word.getLemma());
                    }
                }
                // 写入文件，去重后用逗号分隔
                if (synonyms.size() > 1) {
                    writer.write(String.join(",", synonyms) + "\n");
                }
                indexWord = dictionary.getIndexWordIterator(pos).next();
            }
        }
        writer.close();
    }
}

步骤3：配置Hibernate Search注解 #

把生成的wordnet_synonyms.txt放到资源目录，然后修改你的TokenFilter配置：

@AnalyzerDef(name = "wordnetSynonymAnalyzer",
    tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
    filters = {
        @TokenFilterDef(factory = LowerCaseFilterFactory.class),
        @TokenFilterDef(factory = SynonymFilterFactory.class,
            params = {
                @Parameter(name = "ignoreCase", value = "true"),
                @Parameter(name = "expand", value = "true"),
                @Parameter(name = "synonyms", value = "classpath:wordnet_synonyms.txt")
            })
    })

方案二：自定义同义词提供者动态读取WordNet #

如果需要实时获取WordNet的同义词（比如词库会更新），可以自定义SynonymProvider，让SynonymFilterFactory直接从WordNet查询。

步骤1：实现自定义SynonymProvider #

import net.sf.jwnl.JWNL;
import net.sf.jwnl.data.IndexWord;
import net.sf.jwnl.data.POS;
import net.sf.jwnl.data.Synset;
import net.sf.jwnl.data.Word;
import net.sf.jwnl.dictionary.Dictionary;
import org.apache.lucene.analysis.synonym.SynonymMap;
import org.apache.lucene.analysis.synonym.SynonymProvider;
import org.apache.lucene.util.CharsRef;

import java.io.IOException;
import java.util.*;

public class WordNetSynonymProvider implements SynonymProvider {
    private Dictionary dictionary;

    @Override
    public void init(Map<String, String> params) throws IOException {
        try {
            // 初始化JWNL
            JWNL.initialize(WordNetSynonymProvider.class.getResourceAsStream("/jwnl_properties.xml"));
            dictionary = Dictionary.getInstance();
        } catch (Exception e) {
            throw new IOException("Failed to initialize WordNet dictionary", e);
        }
    }

    @Override
    public SynonymMap build() throws IOException {
        SynonymMap.Builder builder = new SynonymMap.Builder(true);
        // 按需处理词性，这里以名词为例
        processPOS(builder, POS.NOUN);
        processPOS(builder, POS.VERB);
        return builder.build();
    }

    private void processPOS(SynonymMap.Builder builder, POS pos) throws IOException {
        try {
            IndexWord indexWord = dictionary.getIndexWordIterator(pos).next();
            while (indexWord != null) {
                String lemma = indexWord.getLemma();
                Set<String> synonyms = new HashSet<>();
                for (Synset synset : indexWord.getSenses()) {
                    for (Word word : synset.getWords()) {
                        synonyms.add(word.getLemma());
                    }
                }
                synonyms.remove(lemma);
                if (!synonyms.isEmpty()) {
                    // 添加当前词到同义词的映射
                    builder.add(new CharsRef(lemma), new CharsRef[]{new CharsRef(String.join(" ", synonyms))}, true);
                }
                indexWord = dictionary.getIndexWordIterator(pos).next();
            }
        } catch (Exception e) {
            throw new IOException("Failed to process WordNet POS: " + pos, e);
        }
    }
}

步骤2：配置注解使用自定义Provider #

修改@TokenFilterDef，指定synonymsProvider参数替代synonyms文件：

@AnalyzerDef(name = "dynamicWordnetSynonymAnalyzer",
    tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
    filters = {
        @TokenFilterDef(factory = LowerCaseFilterFactory.class),
        @TokenFilterDef(factory = SynonymFilterFactory.class,
            params = {
                @Parameter(name = "ignoreCase", value = "true"),
                @Parameter(name = "expand", value = "true"),
                @Parameter(name = "synonymsProvider", value = "com.yourpackage.WordNetSynonymProvider")
            })
    })

注意事项 #

• WordNet词库准备：需要下载WordNet的数据库文件（比如WordNet 3.1），并在jwnl_properties.xml中指定其路径。
• 性能优化：自定义Provider时建议添加缓存，避免每次构建同义词映射都遍历WordNet，尤其是词库较大时。
• 参数说明：expand=true会让同义词之间互相匹配（比如搜索"car"能匹配"automobile"，搜索"automobile"也能匹配"car"），如果设置为false则是单向映射。

内容的提问来源于stack exchange，提问作者blackhippy