Filebeat插件_meta/fields.yml未加载至Kibana，字段映射异常

问题背景 #

开发自定义Filebeat的pac模块，已编写Ingest Pipeline处理日志，同时通过_meta/fields.yml定义了字段类型（配置见下文），但初始无索引映射时，发送日志后Elasticsearch自动生成的字段类型与fields.yml中的配置不符。

fields.yml配置：

- key: pac
  title: "pac"
  description: Description missing pac Module
  fields:
    - name: pac.log
      type: group
      description: Description missing
      fields:
        - name: deo
          type: group
          description: Description missing
          fields:
            - name: duration
              type: long
              description: Duration of the deo
            - name: category
              type: keyword
              description: Category of the deo
            - name: owner
              type: keyword
              description: User of the deo
            - name: version
              type: float
              description: Version of the deo
            - name: name
              type: keyword
              description: Name of the deo
            - name: reference
              type: double
              description: Referencenumber of the deo
            - name: state
              type: keyword
              description: State of the deo
            - name: status
              type: keyword
              description: Status of the deo
            - name: trigger
              type: group
              description: Description missing
              fields:
                - name: category
                  type: keyword
                  description: Category of the deo-trigger
                - name: name
                  type: text
                  description: Name of the deo-trigger
                - name: path
                  type: text
                  description: Full-Text of the trigger properties
                - name: provider
                  type: keyword
                  description: Supplier of the deo-trigger
            - name: wiring
              type: group
              description: Description missing
              fields:
                - name: async
                  type: boolean
                  description: If deos wired asynchronously
                - name: deoId
                  type: keyword
                  description: Identification number of the deo
                - name: execute
                  type: boolean
                  description: If deo is executed or not
                - name: owner
                  type: keyword
                  description: User of the deo
                - name: shared
                  type: boolean
                  description: Deo was shared
                - name: stopOnError
                  type: boolean
                  description: If deo stopped on Error
        - name: do
          type: group
          description: Description missing
          fields:
            - name: name
              type: keyword
              description: Name of the do task
            - name: state
              type: group
              description: Description missing
              fields:
                - name: from
                  type: keyword
                  description: State from which the do task was used
                - name: to
                  type: keyword
                  description: State to which the do task was switched
        - name: esa
          type: group
          description: Description missing
          fields:
            - name: connection
              type: keyword
              description: Connection status of the ESA
            - name: name
              type: keyword
              description: Name of the ESA
            - name: state
              type: group
              description: Description missing
              fields:
                - name: from
                  type: keyword
                  description: State from which the ESA Connection was established
                - name: to
                  type: keyword
                  description: State to which the ESA Connection was established
        - name: monitor
          type: group
          description: Description missing
          fields:
            - name: heap
              type: group
              description: Description missing
              fields:
                - name: bytes
                  type: long
                  description: Used bytes of the heap
                - name: pct
                  type: float
                  description: Percentage of the maximum available bytes
            - name: heapgc
              type: group
              description: Description missing
              fields:
                - name: bytes
                  type: long
                  description: Used bytes of the heapgc
                - name: pct
                  type: float
                  description: Percentage of the maximum available bytes
        - name: service
          type: group
          description: Description missing
          fields:
            - name: class
              type: keyword
              description: Class of the service
            - name: duration
              type: long
              description: How long the service call took
            - name: name
              type: keyword
              description: Name of the service call
            - name: operation
              type: keyword
              description: Operation type of the service call
            - name: success
              type: boolean
              description: Was the service call successfull or not
        - name: system
          type: group
          description: Description missing
          fields:
            - name: category
              type: keyword
              description: Category of the System
            - name: priority
              type: keyword
              description: Priority of the System
            - name: monitor
              type: group
              description: Description missing
              fields:
                - name: cpu
                  type: group
                  description: Description missing
                  fields:
                    - name: pct
                      type: float
                      description: Percentage of the cpu usage
        - name: wiring
          type: text
          description: Description missing
        - name: meta
          type: text
          description: Description missing
        - name: tags
          type: keyword
          description: Description missing
        - name: timestamp
          type: date
          description: Description missing
        - name: level
          type: keyword
          description: Description missing
        - name: logger
          type: keyword
          description: Description missing

解决步骤 #

1. 修正fields.yml的YAML格式错误 #

原配置中description: Description missing下单独一行的pac Module属于语法错误，会导致YAML解析失败，Filebeat无法加载正确的字段定义。将其合并为单行：

description: Description missing pac Module

2. 确保Filebeat模块结构正确且启用模块 #

• 验证模块路径：确保fields.yml位于modules.d/pac/_meta/目录下（对应你的模块名称）。
• 启用pac模块：在Filebeat配置文件filebeat.yml中添加或确认以下配置：

filebeat.modules:
  - module: pac
    enabled: true
    # 这里添加你的模块输入配置（如日志路径等）

3. 生成并手动加载索引模板 #

Filebeat会根据fields.yml自动生成索引模板，但如果模板未正确同步到Elasticsearch，会导致动态映射接管。可以手动生成并导入：

• 导出模板：

filebeat export template --modules pac --output filebeat-pac-template.json

• 导入到Elasticsearch：

curl -XPUT -H "Content-Type: application/json" http://<你的ES地址>:9200/_index_template/filebeat-pac -d @filebeat-pac-template.json

4. 验证Ingest Pipeline的字段处理逻辑 #

确保Pipeline中对字段的提取、转换操作与fields.yml定义的类型一致：

• 若pac.log.deo.duration定义为long，则Pipeline中不要将其转换为字符串类型。
• 日期类型字段pac.log.timestamp需确保Pipeline正确解析为日期格式，避免ES动态映射识别为字符串。

5. 清理旧索引并重新测试 #

如果之前已经生成了不符合预期的索引，需删除后重新发送日志：

curl -XDELETE http://<你的ES地址>:9200/filebeat-*

重启Filebeat，发送新的日志数据，此时ES会使用导入的模板创建正确的字段映射。

内容的提问来源于stack exchange，提问作者Florian Bär