Elasticsearch设置ignore_above后超长keyword字段写入异常求助

Hey there, let's break down why you're hitting this error and walk through practical solutions that fit your concern about index size.

Why This Error Happens #

You've set ignore_above: 7864320 on your keyword fields, but Elasticsearch (powered by Lucene under the hood) has a hard limit of 32766 bytes per term. Since keyword fields store the entire value as a single term, any content longer than this limit will trigger the max_bytes_length_exceeded_exception—even if ignore_above is set to a higher number. The ignore_above parameter only tells Elasticsearch to skip indexing values longer than the specified length, but it can't bypass Lucene's core term size restriction.

Practical Solutions #

Here are options tailored to your needs (avoiding excessive index size while resolving the error):

1. Store the field without indexing (best if you don't need to search it)

If you only need to retain the full content of PROCESS_INFO.LOG_INFO.MSG and PROCESS_INFO.SERVICE_INFO.EXCEPTION (not search or aggregate on them), set the field type to text with index: false. This way, Elasticsearch stores the full value in _source but doesn't create any index entries, keeping your index size lean.

Update your mapping like this:

"PROCESS_INFO": {
  "properties": {
    "LOG_INFO": {
      "properties": {
        "MSG": {
          "type": "text",
          "index": false
        }
      }
    },
    "SERVICE_INFO": {
      "properties": {
        "EXCEPTION": {
          "type": "text",
          "index": false
        }
      }
    }
  }
}

2. Use a text field with a truncated keyword sub-field (if you need search/aggregation)

If you need to search or aggregate on these fields, combine a text type (which splits content into smaller terms, avoiding the 32766 limit) with a keyword sub-field truncated to 32766 bytes. This gives you full-text search capabilities and limited exact matching without bloating the index too much.

Mapping example:

"PROCESS_INFO": {
  "properties": {
    "LOG_INFO": {
      "properties": {
        "MSG": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 32766
            }
          }
        }
      }
    },
    "SERVICE_INFO": {
      "properties": {
        "EXCEPTION": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 32766
            }
          }
        }
      }
    }
  }
}

• Use MSG (text type) for full-text searches across the content.
• Use MSG.keyword for exact matches on the first 32766 bytes of the content.

3. Truncate content before indexing

If you must keep the field as a keyword, truncate the content to 32766 bytes (about 32KB) before sending it to Elasticsearch. You can do this in two ways:

• In your application code: Trim the long fields before calling the Elasticsearch API.
• Using an Elasticsearch Ingest Pipeline: Create a pipeline to automatically truncate the fields during indexing:

  PUT _ingest/pipeline/truncate_long_fields
  {
    "processors": [
      {
        "truncate": {
          "field": "PROCESS_INFO.LOG_INFO.MSG",
          "length": 32766,
          "encoding": "utf-8"
        }
      },
      {
        "truncate": {
          "field": "PROCESS_INFO.SERVICE_INFO.EXCEPTION",
          "length": 32766,
          "encoding": "utf-8"
        }
      }
    ]
  }
  

  Then specify this pipeline when indexing documents:

  PUT servicerunlogreport-myprogram-2020.07.17/_doc/1?pipeline=truncate_long_fields
  {
    "PROCESS_INFO": {
      "LOG_INFO": {
        "MSG": "your very long message here..."
      }
    }
  }
  

4. Avoid modifying Lucene's global term limit (not recommended)

You could adjust the JVM parameter -Dlucene.max.buffered.bytes to increase the term limit, but this is a global setting that affects all indexes and can lead to increased memory usage and performance degradation. It's not a recommended solution for most production environments.

Final Recommendation #

If you don't need to search these large fields, go with option 1 (text with index: false)—it keeps your index small and avoids the error entirely. If you need search capabilities, option 2 balances functionality and index size perfectly.

内容的提问来源于stack exchange，提问作者jeewonb