EKS环境Loki单实例模式出现empty ring及请求超时问题求助

问题描述 #

在EKS环境以单实例（Monolithic）模式部署Loki数周，创建了数十个Loki Pod，每个Pod接收多台EC2实例日志。Promtail持续返回两类报错：

1. error sending batch, will retry status=500 error="server returned HTTP status 500 Internal Server Error (500): empty ring"
2. error sending batch, will retry status=-1 error="Post \"loki:port/loki/api/v1/push\": context deadline exceeded"

Loki日志对应告警：

level=warn ts=2022-11-07T08:59:39.648738164Z caller=logging.go:86 traceID=414f3905fdec9c5b orgID=fake msg="POST /loki/api/v1/push (500) 5.863871ms Response: \"empty ring\\n\" ws: false; Content-Length: 267464; Content-Type: application/x-protobuf; User-Agent: promtail/2.6.1; X-Amzn-Trace-Id: Root=id; X-Forwarded-For: IP; X-Forwarded-Port: PORT; X-Forwarded-Proto: http; "

level=warn ts=2022-11-07T09:23:23.193157476Z caller=logging.go:86 traceID=0a15f11b53b377ce orgID=fake msg="POST /loki/api/v1/push (500) 9.998004113s Response: \"context canceled\\n\" ws: false; Content-Length: 235297; Content-Type: application/x-protobuf; User-Agent: promtail/2.6.1; X-Amzn-Trace-Id: Root=id; X-Forwarded-For: IP; X-Forwarded-Port: PORT; X-Forwarded-Proto: http; "

故障现象：Grafana调用Loki时断时续，部分Loki Pod短时间内可正常收集日志，随后失败，所有Pod配置基本一致。

核心疑问 #

1. "empty ring"是什么含义？
2. 如何解决当前故障？

附配置文件 #

promtail.yaml #

server:
  http_listen_port: 9080
clients:
  - url: http://ip_where_Loki_run:3100/loki/api/v1/push
positions:
  filename: /usr/local/promtail/positions.yaml
scrape_configs:
  - job_name: server_log
    static_configs:
      - targets:
          - localhost
        labels:
          job: server_log
          hostname: ab
          __path__: /var/log/server.log

Loki.yaml #

auth_enabled: false 

server: 
  http_listen_port: 3100 
  grpc_listen_port: 9096
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600
  http_server_read_timeout: 300s
  http_server_write_timeout: 300s
  http_server_idle_timeout: 300s

ingester: 
  wal: 
    enabled: true 
    dir: /loki/wal 
  lifecycler: 
    ring: 
      kvstore: 
        store: inmemory
      replication_factor: 1 
    final_sleep: 0s 
  chunk_idle_period: 3m       
  chunk_retain_period: 30s
  chunk_encoding: lz4     
  max_transfer_retries: 0     
  chunk_target_size: 1048576  
  max_chunk_age: 1h           

schema_config: 
  configs: 
    - from: 2022-10-05
      store: boltdb-shipper 
      object_store: aws 
      schema: v12 
      index: 
        prefix: index_ 
        period: 24h 

storage_config: 
  boltdb_shipper: 
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: s3 

  aws:
    bucketnames:  bucketnames
    endpoint: s3.us-west-2.amazonaws.com
    region: us-west-2
    access_key_id: access_key_id
    secret_access_key: secret_access_key
    sse_encryption: true

compactor: 
  working_directory: /loki/compactor 
  shared_store: s3 
  compaction_interval: 5m
  retention_enabled: true

limits_config: 
  reject_old_samples: true 
  reject_old_samples_max_age: 720h
  retention_period: 720h
  per_stream_rate_limit: 15MB
  per_stream_rate_limit_burst: 30MB
  ingestion_rate_mb: 15
  ingestion_burst_size_mb: 30

chunk_store_config: 
  max_look_back_period: 0s 

querier:
  query_ingesters_within: 0
  engine:
    max_look_back_period: 3m

query_scheduler:
  max_outstanding_requests_per_tenant: 2048

query_range:
  parallelise_shardable_queries: false
  split_queries_by_interval: 0

frontend:
  max_outstanding_per_tenant: 10240

ingester_client:
  remote_timeout: 30s

解答 #

1. "empty ring"的含义 #

Loki的Ring是组件（如Ingester）集群成员发现的核心机制，所有Ingester节点会加入Ring协同处理日志。"empty ring"表示当前Loki实例的Ring中没有可用的Ingester节点，日志推送请求找不到处理对象，因此返回500错误。

2. 故障原因与解决步骤 #

核心原因

当前Ingester的Ring使用inmemory存储，这是单实例本地内存存储，无法在EKS多Pod集群中共享Ring信息。每个Loki Pod的Ring完全独立，Pod重启或跨Pod交互时就会出现Ring为空的情况；同时Promtail直接指定单个Pod IP，Pod故障时会触发超时。

具体解决措施

• 替换Ring为集群共享存储
  将Ingester配置中的内存存储替换为K8s环境适配的共享发现机制，比如memberlist：

  ingester:
    lifecycler:
      ring:
        kvstore:
          store: memberlist
        replication_factor: 1
  

  同时添加memberlist集群发现配置（需提前创建Loki Headless Service）：

  memberlist:
    join_members:
    - loki-headless.default.svc.cluster.local # 替换为你的Loki Headless Service地址
  

• 修正Promtail的Loki地址
  把Promtail中硬编码的Pod IP替换为Loki的ClusterIP Service地址，避免单个Pod故障导致推送失败：

  clients:
    - url: http://loki-service.default.svc.cluster.local:3100/loki/api/v1/push # 替换为你的Loki Service地址
  

• 调整部署模式
  单实例（Monolithic）模式下多Pod部署会引发组件冲突，建议：

  1. 改用Microservices模式，拆分Ingester、Querier等组件，统一使用共享Ring；
  2. 若坚持单实例模式，限制Pod副本数为1，或确保所有Pod通过共享Ring协同。

• 检查Ingester初始化状态
  查看异常Pod的日志，确认Ingester是否成功启动并加入Ring。若因资源不足导致启动失败，需增加Pod的CPU/内存请求与限制；若因S3存储连接异常，验证IAM权限、endpoint配置是否正确。

内容的提问来源于stack exchange，提问作者Andrew