Filebeat未在OpenSearch中创建索引问题求助

在Windows系统上已安装Filebeat-oss 7.12.0、OpenSearch-2.4.0及OpenSearch Dashboard-2.4.0，所有服务运行正常，但OpenSearch Dashboard中未创建名为filebeat-7.12.0的索引，且无任何报错信息。

相关日志 #

INFO    log/harvester.go:302    Harvester started for file: D:\data\logs.txt
2022-12-08T18:28:17.584+0530    INFO    [crawler]   beater/crawler.go:141   Starting input (ID: 16780016071726099597)
2022-12-08T18:28:17.585+0530    INFO    [crawler]   beater/crawler.go:108   Loading and starting Inputs completed. Enabled inputs: 2
2022-12-08T18:28:17.585+0530    INFO    cfgfile/reload.go:164   Config reloader started
2022-12-08T18:28:17.584+0530    INFO    [input.filestream]  compat/compat.go:111    Input filestream starting
2022-12-08T18:28:17.585+0530    INFO    cfgfile/reload.go:224   Loading of config files completed.
2022-12-08T18:28:20.428+0530    INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:101    add_cloud_metadata: hosting provider type not detected.
2022-12-08T18:28:21.428+0530    INFO    [publisher_pipeline_output] pipeline/output.go:143  Connecting to backoff(elasticsearch(http://localhost:9200))
2022-12-08T18:28:21.428+0530    INFO    [publisher] pipeline/retry.go:219   retryer: send unwait signal to consumer
2022-12-08T18:28:21.428+0530    INFO    [publisher] pipeline/retry.go:223     done
2022-12-08T18:28:21.433+0530    INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 2.4.0
2022-12-08T18:28:21.537+0530    INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 2.4.0
2022-12-08T18:28:21.620+0530    INFO    template/load.go:117    Try loading template filebeat-7.12.0 to Elasticsearch

Filebeat配置文件 #

filebeat.inputs:

- type: log

  
  paths:
    - D:\data\*
    
- type: filestream
  
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - D:\data\*
    
# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false
  
 # ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  
#============================== Kibana =====================================
setup.kibana:
  
  host: "localhost:5601"
  
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  
#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

排查步骤 #

• 检查Filebeat模板是否成功加载
  在OpenSearch Dashboard的Dev Tools中执行以下命令，确认是否存在filebeat-7.12.0模板：

  GET _cat/templates?v
  

  如果未找到该模板，手动执行模板加载命令（Windows PowerShell环境下）：

  .\filebeat.exe setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
  

• 确认Filebeat是否实际发送数据

  1. 查看OpenSearch中的所有索引，确认是否存在带日期后缀的Filebeat索引（默认命名格式为filebeat-7.12.0-YYYY.MM.DD）：

     GET _cat/indices?v
     

  2. 检查D:\data下的日志文件是否有新内容写入（Filebeat默认仅读取新增内容）；若文件已被读取过，删除Filebeat目录下的data/registry/filebeat/registry文件，重启Filebeat重新读取文件。

• 检查OpenSearch索引自动创建设置
  打开OpenSearch的config/opensearch.yml配置文件，确认action.auto_create_index参数是否允许创建Filebeat相关索引。若设置了特定值，需包含filebeat-*，例如：

  action.auto_create_index: filebeat-*,*
  

  若该参数为false，则需手动创建索引模板及索引。

• 修正Filebeat配置中的重复输入
  配置中同时启用了log（已弃用）和filestream两种输入类型，且指向同一目录，易导致冲突。建议仅保留filestream输入，修改后重启Filebeat：

  filebeat.inputs:
  - type: filestream
    enabled: true
    paths:
      - D:\data\*
  

• 验证权限配置

  1. 确保运行Filebeat的用户拥有D:\data目录的读取权限；
  2. 若OpenSearch启用了安全插件，检查Filebeat使用的用户是否具备create_index和write权限，可在Dev Tools中执行：

     GET _security/user/_current
     

• 检查版本兼容性
  Filebeat 7.12.0与OpenSearch 2.4.0版本跨度较大，存在兼容性风险。建议更换为与OpenSearch版本更匹配的采集工具，例如OpenSearch Beat或Elasticsearch 7.17.x版本的Filebeat（OpenSearch 2.x基于Elasticsearch 7.17分支开发）。

内容的提问来源于stack exchange，提问作者Anish Mittal