ES 6.8迁移ES2.4含stopFilter与synonymFilter的索引配置失败怎么办？

问题描述 #

在 Elasticsearch 2.4 中有一个索引使用如下分析配置：

{"settings":{"index":{"analysis":{"filter":{"stopFilter":{"type":"stop","remove_trailing":"false","stopwords":"_english_"},"stemmerFilter":{"type":"stemmer","language":"light_english"},"synonymFilter":{"type":"synonym","synonyms":["3pip, third party checkout with immediate => 3PIP","b to b, business to business, btob, b2b, business 2 business, b 2 b => B2B","b 2 e, business to employee, biz to employee, btoe, b to e, b2e => B2E","b 2 g, b to g, btog, b2g => B2G"]},"possessiveFilter":{"type":"stemmer","language":"possessive_english"}},"char_filter":{"quotes":{"type":"mapping","mappings":["\\u0091=>\\u0027","\\u0092=>\\u0027","\\u2018=>\\u0027","\\u2019=>\\u0027","\\u201B=>\\u0027"]}},"analyzer":{"default":{"filter":["possessiveFilter","lowercase","stopFilter","stemmerFilter","synonymFilter","asciifolding"],"char_filter":["html_strip","quotes"],"type":"custom","tokenizer":"standard"},"lowercase":{"filter":["lowercase"],"type":"custom","tokenizer":"keyword"},"autocomplete":{"filter":["possessiveFilter","lowercase","stopFilter","stemmerFilter","synonymFilter","asciifolding"],"char_filter":["html_strip","quotes"],"type":"custom","tokenizer":"autocompleteTokenizer"},"autocomplete_search":{"filter":["lowercase","stopFilter","stemmerFilter","synonymFilter","asciifolding"],"tokenizer":"standard"}},"tokenizer":{"autocompleteTokenizer":{"token_chars":["letter","digit"],"min_gram":"2","type":"edgeNGram","max_gram":"20"}}}}}}

将该配置直接迁移到 Elasticsearch 6.8 时创建索引失败，报错信息如下：

"type":"illegal_argument_exception","reason":"failed to build synonyms","caused_by":{"type":"parse_exception","reason":"parse_exception: Invalid synonym rule at line 5","caused_by":{"type":"illegal_argument_exception","reason":"term: a b a analyzed to a token (b) with position increment != 1 (got: 2)"}},"status":400

以及：

"reason":"term: third party checkout with immediate analyzed to a token (immediate) with position increment != 1 (got: 2)"

移除同义词规则中包含停用词的短语可以创建索引，但这并非可行方案，需要找到在 ES 6.8 中保留原配置（同时使用 stopFilter 和 synonymFilter）的方法。

解决办法 #

这个问题是由于停用词过滤器过滤掉同义词短语中的停用词后，导致 token 位置增量不连续，触发了 ES 6.8 同义词过滤器的严格校验。以下两种方案可以解决该问题：

方案1：开启同义词过滤器的宽松模式 #

在 synonymFilter 的配置中添加 "lenient": true 参数，让 ES 忽略位置增量不连续的异常，从而正常创建索引。修改后的同义词过滤器配置如下：

"synonymFilter": {
  "type": "synonym",
  "lenient": true,
  "synonyms": [
    "3pip, third party checkout with immediate => 3PIP",
    "b to b, business to business, btob, b2b, business 2 business, b 2 b => B2B",
    "b 2 e, business to employee, biz to employee, btoe, b to e, b2e => B2E",
    "b 2 g, b to g, btog, b2g => B2G"
  ]
}

方案2：调整过滤器执行顺序 #

将 synonymFilter 的执行顺序提前到 stopFilter 之前，这样同义词规则会先处理原始文本，再由停用词过滤器过滤停用词，避免位置增量异常。以 default 分析器为例，修改后的配置如下：

"default": {
  "filter": ["possessiveFilter", "lowercase", "synonymFilter", "stopFilter", "stemmerFilter", "asciifolding"],
  "char_filter": ["html_strip", "quotes"],
  "type": "custom",
  "tokenizer": "standard"
}

需要同步修改 autocomplete 和 autocomplete_search 分析器的过滤器顺序，确保逻辑一致。

内容的提问来源于stack exchange，提问作者Sumit Kumar