运行Elasticsearch全文查询时遇「standard分词过滤器已移除」错误求助

问题场景 #

运行以下全字段全文搜索查询时，开发服务器正常执行，但生产环境抛出异常：

GET /MyIndex_2/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "test"
          }
        }
      ]
    }
  }
}

报错信息 #

"The [standard] token filter has been removed"

关联索引配置 #

生产环境使用的索引配置如下：

PUT analyzed_accounts_consolidated10052020_azure_prod4
{
  "settings": {
    "index": {
      "number_of_shards": "5",
      "max_result_window": "10000000",
      "analysis": {
        "filter": {
          "synonym_filter_cvtext": {
            "ignore_case": "true",
            "expand": "true",
            "type": "synonym",
            "synonyms_path": "ho2s_filters/synonyms_cvtext.txt"
          },
          "stemmer_it": {
            "type": "stemmer",
            "language": "light_italian"
          },
          "synonym_filter_skills_roles": {
            "ignore_case": "true",
            "expand": "true",
            "type": "synonym",
            "synonyms_path": "ho2s_filters/synonyms_skills_roles.txt"
          },
          "stopwords_it": {
            "type": "stop",
            "stopwords_path": "ho2s_filters/stopwords.txt"
          },
          "synonym_filter_educations": {
            "ignore_case": "true",
            "expand": "true",
            "type": "synonym",
            "synonyms_path": "ho2s_filters/synonyms_educations.txt"
          }
        },
        "analyzer": {
          "analyzer_skills_roles": {
            "filter": [
              "lowercase",
              "synonym_filter_skills_roles",
              "stopwords_it",
              "stemmer_it"
            ],
            "char_filter": [
              "mapping_skills_roles"
            ],
            "type": "custom",
            "tokenizer": "standard"
          },
          "case_insensitive_sort": {
            "filter": [
              "standard",
              "lowercase",
              "trim"
            ],
            "type": "custom",
            "tokenizer": "keyword"
          },
          "analyzer_cvtext": {
            "filter": [
              "lowercase",
              "synonym_filter_cvtext",
              "stopwords_it",
              "stemmer_it"
            ],
            "char_filter": [
              "mapping_skills_roles"
            ],
            "type": "custom",
            "tokenizer": "standard"
          },
          "analyzer_educations": {
            "filter": [
              "lowercase",
              "synonym_filter_educations",
              "stopwords_it",
              "stemmer_it"
            ],
            "char_filter": [
              "mapping_skills_roles"
            ],
            "type": "custom",
            "tokenizer": "standard"
          }
        },
        "char_filter": {
          "mapping_skills_roles": {
            "type": "mapping",
            "mappings_path": "ho2s_filters/mapping_skills_roles.txt"
          }
        }
      },
      "number_of_replicas": "1"
    }
  }
}

问题原因 #

Elasticsearch 7.0及以上版本移除了standard token filter，开发环境使用的ES版本较低（仍支持该配置），生产环境版本更高，导致配置不兼容抛出异常。异常触发点是索引配置中case_insensitive_sort分析器的filter列表里包含了已被移除的standard过滤器。

解决方案 #

1. 修改索引分析器配置
   将case_insensitive_sort分析器的filter数组中的"standard"项移除，修改后的配置如下：

   "case_insensitive_sort": {
     "filter": [
       "lowercase",
       "trim"
     ],
     "type": "custom",
     "tokenizer": "keyword"
   }
   

   注：该分析器使用keyword tokenizer，不会拆分文本，standard过滤器在此场景下无实际作用，移除后不影响原有功能

2. 重新部署索引
   由于索引创建后无法直接修改分析器配置，需按以下步骤操作：

   • 备份原索引的所有数据
   • 使用修改后的配置创建新索引
   • 将备份的数据导入新索引
   • 切换业务查询到新索引，验证查询是否正常执行

内容的提问来源于stack exchange，提问作者Code Noob