ELK索引滚动更新后日期未变更，如何修改命名规则？

问题现象 #

使用ELK（elasticsearch-8.12.0-1.x86_64）存储Kong API网关日志，通过ILM管理索引生命周期，但新创建的索引始终使用固定日期后缀命名，示例如下：

kong-2022-11-17-000001
kong-2022-11-17-000002
kong-2022-11-17-000003
kong-2022-11-17-000004
kong-2022-11-17-000005
kong-2022-11-17-000006

期望实现按索引创建日期命名，示例如下：

kong-2022-11-17-000001
kong-2022-11-17-000002
kong-2022-11-17-000003
kong-2022-12-25-000001
kong-2023-01-01-000001

现有配置 #

Logstash管道配置（/etc/logstash/kong.conf） #

elasticsearch {
    hosts => ["https://elastic01:elastic_port" , "https://elastic02:elastic_port" , "https://elastic03:elastic_port"]
    user => "elastic_user"
    password => elastic_user_password
    ssl => true
    ssl_certificate_verification => false
    cacert => "/etc/logstash/http_ca.crt"
    ilm_rollover_alias => "kong"
    ilm_pattern => "{now/d}-000001"
    ilm_policy => "kong-index-policy"
}

索引模板（kong-index-template） #

{
  "index": {
    "lifecycle": {
      "name": "kong-index-policy",
      "rollover_alias": "kong"
    },
    "mapping": {
      "total_fields": {
        "limit": "10000"
      }
    },
    "refresh_interval": "5s"
  }
}

ILM策略（kong-index-policy） #

{
  "policy": "kong-index-policy",
  "phase_definition": {
    "min_age": "0ms",
    "actions": {
      "rollover": {
        "max_age": "180d",
        "max_primary_shard_size": "10gb"
      },
      "set_priority": {
        "priority": 100
      }
    }
  }
}

已尝试操作及报错 #

尝试创建带日期的写入索引：

PUT %3Ckong-%7Bnow%2Fd%7D-000001%3E
{
 "aliases": {
   "kong": {
     "is_write_index": true
   }
 }
}

返回错误：

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason": "别名 [kong] 存在多个写入索引 [kong-2024.06.05-000001,kong-2022-11-24-000009]"
      }
    ],
    "type": "illegal_state_exception",
    "reason": "别名 [kong] 存在多个写入索引 [kong-2024.06.05-000001,kong-2022-11-24-000009]"
  },
  "status": 500
}

随后执行命令将旧索引的写入状态设为false：

POST /_aliases
{
  "actions": [
    {
      "add": {
        "index": "kong-2022-11-24-000009",
        "alias": "kong",
        "is_write_index": false
      }
    }]
}

解决方案 #

1. 修正Logstash配置 #

修改Logstash管道配置，禁用自动模板管理并指定自定义模板，避免Logstash覆盖模板设置：

elasticsearch {
    hosts => ["https://elastic01:elastic_port" , "https://elastic02:elastic_port" , "https://elastic03:elastic_port"]
    user => "elastic_user"
    password => elastic_user_password
    ssl => true
    ssl_certificate_verification => false
    cacert => "/etc/logstash/http_ca.crt"
    ilm_rollover_alias => "kong"
    ilm_pattern => "{now/d}-000001"
    ilm_policy => "kong-index-policy"
    manage_template => false  # 禁用自动模板管理
    template_name => "kong-index-template"  # 指定使用自定义模板
}

2. 更新索引模板 #

确保模板匹配所有kong前缀的索引，并设置足够高的优先级：

PUT _index_template/kong-index-template
{
  "index_patterns": ["kong-*"],
  "template": {
    "index": {
      "lifecycle": {
        "name": "kong-index-policy",
        "rollover_alias": "kong"
      },
      "mapping": {
        "total_fields": {
          "limit": "10000"
        }
      },
      "refresh_interval": "5s"
    }
  },
  "priority": 100
}

3. 调整ILM滚动策略 #

若需每天自动创建新索引，将ILM的滚动条件max_age改为1天（原配置为180天，仅在索引超大或超期180天才滚动）：

PUT _ilm/policy/kong-index-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_primary_shard_size": "10gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "180d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

4. 初始化正确的写入索引 #

确保当前只有一个写入索引，创建符合日期命名规则的初始索引：

PUT kong-{{now/d}}-000001
{
  "aliases": {
    "kong": {
      "is_write_index": true
    }
  }
}

注：若使用curl执行，需将{{now/d}}编码为%7Bnow%2Fd%7D。

5. 重启Logstash服务 #

使配置生效：

systemctl restart logstash

验证方法 #

• 手动触发滚动测试：

  POST /kong/_rollover
  

• 查看索引列表，确认新索引命名格式：

  GET _cat/indices/kong-*
  

• 检查别名的写入索引状态：

  GET _alias/kong
  

内容的提问来源于stack exchange，提问作者Linux_Admin