Elasticsearch7.9升级：已弃用PatternAnalyzer类的替代方案求助

背景 #

我们正将Elasticsearch从5.6版本升级至7.9版本，原代码中使用的org.elasticsearch.index.analysis.PatternAnalyzer类已被弃用。尝试替换为PatternTokenizer后出现编译错误，请求提供正确的替代实现方案。

Gradle依赖 #

compile 'org.elasticsearch.client:elasticsearch-rest-client:7.9.0'
compile 'org.elasticsearch:elasticsearch:7.9.0'

原Tokenizer.java代码片段 #

import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import org.apache.lucene.analysis.util.CharArraySet;
import org.elasticsearch.index.analysis.PatternAnalyzer;

private static Iterable<String> splitString(String str) {
       PatternAnalyzer analyzer = new PatternAnalyzer(
               Pattern.compile("(\\s+|-+)"), true, new CharArraySet(10, false));

       List<String> results = new ArrayList<String>();
       String[] initialTokens = str.split("\\s+");
       for (String token : initialTokens) {
           if (token.matches("\\d+\\w*\\-\\d+\\w*")) {
               results.add(token);
           } else {
               try {
                   TokenStream tokenStream = analyzer.tokenStream(token, token);
                   tokenStream.reset();
                   CharTermAttribute termAttribute = tokenStream.getAttribute(CharTermAttribute.class);

                   while (tokenStream.incrementToken()) {
                       results.add(termAttribute.toString());
                   }

                   tokenStream.end();
                   tokenStream.close();

               } catch (IOException e) {
                   throw new RuntimeException(e);
               }
           }
       }

       analyzer.close();
       return results;
   }

原编译错误 #

Tokenizer.java:13: error: cannot find symbol
import org.apache.lucene.analysis.util.CharArraySet;
                                      ^
  symbol:   class CharArraySet
  location: package org.apache.lucene.analysis.util

Tokenizer.java:14: error: cannot find symbol
import org.elasticsearch.index.analysis.PatternAnalyzer;
                                       ^
  symbol:   class PatternAnalyzer
  location: package org.elasticsearch.index.analysis

Tokenizer.java:157: error: cannot find symbol
        PatternAnalyzer analyzer = new PatternAnalyzer(
        ^
  symbol:   class PatternAnalyzer
  location: class Tokenizer

Tokenizer.java:157: error: cannot find symbol
        PatternAnalyzer analyzer = new PatternAnalyzer(
                                       ^
  symbol:   class PatternAnalyzer
  location: class Tokenizer

Tokenizer.java:158: error: cannot find symbol
                Pattern.compile("(\\s+|-+)"), true, new CharArraySet(10, false));
                                                        ^
  symbol:   class CharArraySet
  location: class Tokenizer

替换为PatternTokenizer后的代码片段 #

import org.apache.lucene.analysis.CharArraySet;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.pattern.PatternTokenizer;

private static Iterable<String> splitString(String str) {
       PatternTokenizer analyzer = new PatternTokenizer(
               Pattern.compile("(\\s+|-+)"), true, new CharArraySet(10, false));

       List<String> results = new ArrayList<String>();
       String[] initialTokens = str.split("\\s+");
       for (String token : initialTokens) {
           if (token.matches("\\d+\\w*\\-\\d+\\w*")) {
               results.add(token);
           } else {
               try {
                   TokenStream tokenStream = analyzer.tokenStream(token, token);
                   tokenStream.reset();
                   CharTermAttribute termAttribute = tokenStream.getAttribute(CharTermAttribute.class);

                   while (tokenStream.incrementToken()) {
                       results.add(termAttribute.toString());
                   }

                   tokenStream.end();
                   tokenStream.close();

               } catch (IOException e) {
                   throw new RuntimeException(e);
               }
           }
       }

       analyzer.close();
       return results;
   }

新编译错误 #

Tokenizer.java:15: error: cannot find symbol
    import org.apache.lucene.analysis.util.CharArraySet;
                                          ^
      symbol:   class CharArraySet
      location: package org.apache.lucene.analysis.util
    
    Tokenizer.java:16: error: cannot find symbol
    import org.elasticsearch.index.analysis.PatternAnalyzer;
                                           ^
      symbol:   class PatternAnalyzer
      location: package org.elasticsearch.index.analysis
    
    Tokenizer.java:160: error: incompatible types: Pattern cannot be converted to AttributeFactory
                    Pattern.compile("(\\s+|-+)"), true, new CharArraySet(10, false));
                                   ^
    Tokenizer.java:169: error: cannot find symbol
                        TokenStream tokenStream = analyzer.tokenStream(token, token);
                                                          ^
      symbol:   method tokenStream(String,String)
      location: variable analyzer of type PatternTokenizer

正确的替代实现方案 #

问题核心 #

1. PatternAnalyzer是Elasticsearch封装的完整分析器，7.x版本已移除；而PatternTokenizer只是Lucene的文本拆分器，不具备分析器的完整能力。
2. Lucene 7.x中CharArraySet的包路径从org.apache.lucene.analysis.util迁移到了org.apache.lucene.analysis.core。
3. 原代码中PatternAnalyzer的逻辑是：正则拆分文本 + 转小写 + 忽略停用词，需要用Lucene的基础组件重新组合实现。

修正后的代码 #

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.core.CharArraySet;
import org.apache.lucene.analysis.core.LowerCaseFilter;
import org.apache.lucene.analysis.pattern.PatternTokenizer;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.List;
import java.util.regex.Pattern;

private static Iterable<String> splitString(String str) {
    // 自定义分析器，还原原PatternAnalyzer的核心逻辑：正则拆分+转小写
    Analyzer analyzer = new Analyzer() {
        @Override
        protected TokenStreamComponents createComponents(String fieldName) {
            // PatternTokenizer参数：拆分正则、-1表示提取分隔符之外的内容
            PatternTokenizer tokenizer = new PatternTokenizer(Pattern.compile("(\\s+|-+)"), -1);
            // 添加转小写过滤器，对应原PatternAnalyzer的lowercase=true参数
            TokenStream stream = new LowerCaseFilter(tokenizer);
            return new TokenStreamComponents(tokenizer, stream);
        }
    };

    List<String> results = new ArrayList<>();
    String[] initialTokens = str.split("\\s+");
    for (String token : initialTokens) {
        if (token.matches("\\d+\\w*\\-\\d+\\w*")) {
            results.add(token);
        } else {
            // 使用try-with-resources自动关闭TokenStream，简化资源管理
            try (TokenStream tokenStream = analyzer.tokenStream("", new StringReader(token))) {
                tokenStream.reset();
                CharTermAttribute termAttribute = tokenStream.addAttribute(CharTermAttribute.class);
                while (tokenStream.incrementToken()) {
                    results.add(termAttribute.toString());
                }
                tokenStream.end();
            } catch (IOException e) {
                throw new RuntimeException(e);
            }
        }
    }

    analyzer.close();
    return results;
}

关键修正点 #

• 用Lucene的Analyzer抽象类自定义分析器，内部组合PatternTokenizer和LowerCaseFilter，完全还原原PatternAnalyzer的行为。
• 修正CharArraySet的导入路径为org.apache.lucene.analysis.core.CharArraySet。
• 调整PatternTokenizer的构造参数，传入-1确保提取分隔符之外的有效文本内容。
• 使用StringReader作为文本输入，适配Lucene 7.x的tokenStream方法参数要求。
• 用try-with-resources自动管理TokenStream资源，避免手动关闭的繁琐和遗漏。

内容的提问来源于stack exchange，提问作者LostDev