Loki查询11分钟内日志正常，超12分钟查询出现请求超时错误

通过Docker Compose部署Loki、Grafana、Promtail容器环境，短时间范围日志查询正常，但扩大查询时间范围后出现超时错误：

net/http: request canceled (Client.Timeout exceeded while awaiting headers)

部署配置 #

docker-compose.yaml #

version: "3"

networks:
  loki:
    driver: bridge
    ipam:
      config: 
        - subnet: 192.180.2.0/24

services:
  loki:
    image: grafana/loki:2.8.6
    container_name: loki
    volumes:
      - ./loki-config.yaml:/etc/loki/loki-config.yaml
      - /etc/localtime:/etc/localtime:ro
      - /data:/tmp
    command: -config.file=/etc/loki/loki-config.yaml
    ports:
      - "3100:3100"
    networks:
      - loki
    logging:
      driver: "json-file"
      options:
        max-size: 1m
        max-file: "3"
        tag: "common.logging.loki.docker"

  promtail:
    image: grafana/promtail:2.4.2
    container_name: promtail
    restart: always
    volumes:
      - /var/lib/docker/:/var/lib/docker:ro 
      - /var/log:/var/log
      - ./promtail-local-config.yaml:/etc/promtail/config.yaml:ro
      - /etc/localtime:/etc/localtime:ro
    command: -config.file=/etc/promtail/config.yaml
    networks:
      - loki
    ports:
      - "1514:1514/tcp"
      - "9080:9080/tcp"
    logging:
      driver: "json-file"
      options:
        max-size: 1m
        max-file: "3"
        tag: "common.logging.promtail.docker"

  grafana:
    image: grafana/grafana:10.2.0
    container_name: grafana
    restart: always
    environment:
      - GF_DATAPROXY_TIMEOUT=1200
      - GF_DATAPROXY_KEEP_ALIVE_SECONDS=300
      - GF_LOG_LEVEL=debug
    ports:
      - "3000:3000"
    networks:
      - loki
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /opt/grafana_data:/var/lib/grafana
      - /opt/grafana_etc:/etc/grafana
    logging:
      driver: "json-file"
      options:
        max-size: 1m
        max-file: "3"
        tag: "common.logging.grafana.docker"

loki-config.yaml #

auth_enabled: false

server:
  http_listen_port: 3100

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v12
      index:
        prefix: index_
        period: 24h

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules

  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

limits_config:
  split_queries_by_interval: 15m
  max_entries_limit_per_query: 15000

query_range:
  # make queries more cache-able by aligning them with their step intervals
  align_queries_with_step: true
  max_retries: 5
  parallelise_shardable_queries: true
  cache_results: true

ruler:
  storage:
    type: local
    local:
      directory: /tmp/rules

查询表现：最近5分钟日志（4200条）可在0.2秒返回，6-12分钟范围查询正常，但更大时间范围查询触发超时。

• Loki服务端超时未配置：当前Loki的server块未设置HTTP处理超时，大时间范围查询耗时超出客户端（Grafana）等待阈值。需添加服务端超时配置：

  server:
    http_listen_port: 3100
    http_server_timeout: 300s  # 延长服务端处理请求的超时时间
  

• 查询条目上限过低：limits_config中的max_entries_limit_per_query设为15000，大时间范围查询可能触发该限制，导致Loki处理中断或延迟。可根据实际需求调高该值：

  limits_config:
    split_queries_by_interval: 15m
    max_entries_limit_per_query: 50000  # 提升单查询返回条目上限
  

• 存储IO性能瓶颈：使用本地文件系统存储时，大查询需读取大量chunk文件，磁盘IO不足会拖慢处理速度。可检查磁盘IO使用率，考虑更换高速存储介质，或优化存储缓存策略。
• 组件版本不兼容：Promtail版本2.4.2远低于Loki的2.8.6，版本差异可能导致日志存储格式不匹配，大查询时出现异常。将Promtail升级至与Loki相同的2.8.6版本。
• Grafana数据源超时未生效：尽管设置了GF_DATAPROXY_TIMEOUT=1200，仍需确认Grafana中Loki数据源的单独超时配置（在数据源页面可调整），同时排查是否存在反向代理、浏览器等其他层级的超时限制。

内容的提问来源于stack exchange，提问作者Šimon Kovčák