从ElasticSearch迁移至OpenSearch时Must子句解析异常问题

报错信息 #

{"message": "x_content_parse_exception: [parsing_exception] Reason: unknown query [must]","data": {}}

问题分析 #

控制台打印显示mustClauses被错误赋值为对象而非数组，导致后续push操作异常，最终引发请求超时。原代码处理时间范围时，错误地将mustClauses从数组替换成了包含must键的对象，而OpenSearch的bool查询要求must的值是查询条件数组。

原错误DSL代码 #

let query;

const { startDate, endDate } = calculateTimeRange(time);
let mustClauses : any = []; 
let mustNot : any = [];
  if(startDate && endDate){
    mustClauses = {
      "must" : {
        "range" : {
          "offense_start_time" : {
            "gte" : startDate,
            "lte" : endDate
          }
        }
      }
    }
  }
  if(type !== "ALL"){
    mustClauses.push({
      "term" : {
        "source_siem.keyword" : type
      }
    })
  }
  if(telco_type !=="ALL"){
      if(telco_type=="telco"){
          mustNot = {
            "must_not" : {
              "term" : {
                "telco_name.keyword": this.configService.get("LOCAL_TELCO")
              }
            }
          }
      }else{
        mustClauses.push({
          "term" : {
            "telco_name.keyword" : telco_type
          }
        })
      }
  }
if (select_records_by_type == "0" || select_records_by_type == "ALL" || select_records_by_type == 0) {
  query = {
    "track_total_hits" : true,
    "size": 0,
    "query": {
      "bool": {
        "must": mustClauses,
        "must_not": mustNot || []
      }
    },
    "aggs": {
      "group_by_<base_column>": {
        "terms": {
          "field": "<base_column>.keyword",
          "size": 10000
        }
      }
    }
  };

原正常运行的ElasticSearch SQL查询 #

query = `SELECT ${base_column}, count(*) as frequency
          FROM ${this.configService.get("ALERT_INDEX")}
          WHERE ${type !== "ALL" ? `source_siem='${type}'` : "1=1"}
          AND ${ 
             startDate != undefined && endDate != undefined
             ? `offense_start_time >= ${startDate} AND offense_start_time < ${endDate}`
             : "1=1"
           } 
          AND ${telco_type !== "ALL" ? (telco_type == "telco" ? `telco_name !='${this.configService.get("LOCAL_TELCO")}'` : `telco_name='${telco_type}'`) : "1=1"}
          GROUP BY ${base_column}`;

控制台输出 #

-------------------I am in the offence service-----------------------
-----------------------I am in the new conversion---------------------------------
{ must: { range: { offense_start_time: [Object] } } }
query in here
{
  track_total_hits: true,
  size: 0,
  query: { bool: { must: [Object], must_not: [] } },
  aggs: { 'group_by_<base_column>': { terms: [Object] } }
}

修正方案 #

核心错误点：

1. 处理时间范围时，不应替换mustClauses为对象，而是向数组中pushrange查询条件
2. mustNot同样应该维护为数组，而非包含must_not键的对象

修正后的代码：

let query;

const { startDate, endDate } = calculateTimeRange(time);
let mustClauses : any = []; 
let mustNot : any = [];

// 修正：向数组push range条件，而非替换整个数组为对象
if(startDate && endDate){
  mustClauses.push({
    "range" : {
      "offense_start_time" : {
        "gte" : startDate,
        "lte" : endDate
      }
    }
  });
}

if(type !== "ALL"){
  mustClauses.push({
    "term" : {
      "source_siem.keyword" : type
    }
  });
}

if(telco_type !=="ALL"){
  if(telco_type=="telco"){
    // 修正：向mustNot数组push条件，而非赋值为对象
    mustNot.push({
      "term" : {
        "telco_name.keyword": this.configService.get("LOCAL_TELCO")
      }
    });
  }else{
    mustClauses.push({
      "term" : {
        "telco_name.keyword" : telco_type
      }
    });
  }
}

if (select_records_by_type == "0" || select_records_by_type == "ALL" || select_records_by_type == 0) {
  query = {
    "track_total_hits" : true,
    "size": 0,
    "query": {
      "bool": {
        // must和must_not直接使用数组即可，不需要额外嵌套键
        "must": mustClauses.length > 0 ? mustClauses : undefined,
        "must_not": mustNot.length > 0 ? mustNot : undefined
      }
    },
    "aggs": {
      "group_by_<base_column>": {
        "terms": {
          "field": "<base_column>.keyword",
          "size": 10000
        }
      }
    }
  };
}

修正说明 #

• OpenSearch的bool查询中，must、must_not、should等参数的值必须是查询条件数组，而非包含对应键的对象
• 原代码错误地将mustClauses从数组替换为{ must: {...} }，导致OpenSearch解析时识别到未知的must查询类型
• mustNot同理，应维护为数组，直接传入bool的must_not字段

内容的提问来源于stack exchange，提问作者Muhammad Muneeb Waqas