Filebeat创建的索引不遵循Index Template映射问题排查

问题 #

使用Filebeat 7.17.22版本时，已配置Index Template及映射规则，但Filebeat创建的索引未遵循该映射。例如pm2_env.axm_monitor.Event Loop Latency.value字段在模板中设置为double类型，但实际索引中却被映射为text类型。

Filebeat配置（filebeat.yml） #

setup.dashboards.enabled: false
setup.ilm.enabled: true
setup.ilm.pattern: '{now/d}-000001'
setup.ilm.policy_name: 1d_rollover_90d_coldstorage_365d_delete
setup.ilm.rollover_alias: filebeat-logs-ov2-staging-microservices
setup.template.enabled: true
setup.template.name: filebeat-logs-ov2-staging-microservices
setup.template.pattern: filebeat-logs-ov2-staging-microservices-*
setup.template.settings:
  index.lifecycle.name: 1d_rollover_90d_coldstorage_365d_delete
  index.lifecycle.rollover_alias: filebeat-logs-ov2-staging-microservices

自定义Index Template映射 #

{
  "properties": {
    "geoip": {
      "properties": {
        "location": {
          "type": "geo_point"
        }
      }
    },
    "pm2_env": {
      "properties": {
        "axm_monitor": {
          "properties": {
            "Event Loop Latency": {
              "properties": {
                "value": {
                  "type": "double"
                }
              }
            },
            "Used Heap Size": {
              "properties": {
                "value": {
                  "type": "double"
                }
              }
            },
            "Event Loop Latency p95": {
              "properties": {
                "value": {
                  "type": "double"
                }
              }
            },
            "Heap Size": {
              "properties": {
                "value": {
                  "type": "double"
                }
              }
            }
          }
        }
      }
    },
    "responseSize": {
      "type": "double"
    },
    "processingTime": {
      "type": "double"
    }
  }
}

实际索引中的错误映射 #

"Event Loop Latency": {
  "properties": {
    "historic": {
      "type": "boolean"
    },
    "type": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "unit": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "value": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    }
  }
}

排查与解决思路 #

1. 调整自定义模板优先级 #

Filebeat自带默认模板，若你的自定义模板优先级低于默认模板（默认优先级通常为100），会被覆盖。

• 执行命令查看所有匹配模板：

  GET _index_template
  

• 修改自定义模板优先级为更高值（比如200）：

  PUT _index_template/filebeat-logs-ov2-staging-microservices
  {
    "index_patterns": ["filebeat-logs-ov2-staging-microservices-*"],
    "priority": 200,
    "template": {
      "mappings": {
        // 你的自定义映射内容
      }
    }
  }
  

2. 确保Filebeat加载自定义映射 #

当前配置未指定自定义映射的加载方式，需在filebeat.yml中明确配置映射：

setup.template.enabled: true
setup.template.name: filebeat-logs-ov2-staging-microservices
setup.template.pattern: filebeat-logs-ov2-staging-microservices-*
setup.template.mappings:
  properties:
    geoip:
      properties:
        location:
          type: geo_point
    pm2_env:
      properties:
        axm_monitor:
          properties:
            "Event Loop Latency":
              properties:
                value:
                  type: double
            "Used Heap Size":
              properties:
                value:
                  type: double
            "Event Loop Latency p95":
              properties:
                value:
                  type: double
            "Heap Size":
              properties:
                value:
                  type: double
    responseSize:
      type: double
    processingTime:
      type: double

修改后重新加载模板：

filebeat setup --template -e

3. 阻止动态映射覆盖规则 #

若Elasticsearch动态映射开启，首次写入的字符串类型value会自动生成错误映射，后续模板无法覆盖：

• 删除已有错误索引和别名：

  DELETE filebeat-logs-ov2-staging-microservices-*
  DELETE _alias/filebeat-logs-ov2-staging-microservices
  

• 在自定义模板中添加dynamic: false，关闭目标字段的动态映射：

  "pm2_env": {
    "properties": {
      "axm_monitor": {
        "dynamic": false,
        "properties": {
          "Event Loop Latency": {
            "properties": {
              "value": {
                "type": "double"
              }
            }
          }
          // 其他字段映射...
        }
      }
    }
  }
  

4. 验证ILM与模板的关联 #

确认ILM策略关联的别名和模板一致：

GET _ilm/policy/1d_rollover_90d_coldstorage_365d_delete

确保策略中的rollover_alias与模板配置完全匹配，新创建的索引会正确应用自定义模板。

内容的提问来源于stack exchange，提问作者Terchila Marian