Loki 2.8.2对接MinIO(S3)无索引/块目录生成问题求助

问题背景 #

在独立虚拟机上通过Docker Compose部署Loki 2.8.2，使用MinIO作为S3兼容对象存储。MinIO桶已配置公开读写权限，Loki日志无报错，已生成loki_cluster_seed.json文件，但MinIO桶内始终未出现index/chunks目录，无法将日志存储到S3中。

现有配置 #

Docker Compose配置 #

---
version: "3"

networks:
  loki:

services:
  read:
    image: grafana/loki:2.8.2
    command: "-config.file=/etc/loki/config.yaml -target=read -config.expand-env=true -print-config-stderr log-config-reverse-order"
    ports:
      - 3101:3100
      - 7946
      - 9095
    volumes:
      - ./loki-config.yaml:/etc/loki/config.yaml
    healthcheck:
      test: [ "CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3100/ready || exit 1" ]
      interval: 10s
      timeout: 5s
      retries: 5
    networks: &loki-dns
      loki:
        aliases:
          - loki

  write:
    image: grafana/loki:2.8.2
    command: "-config.file=/etc/loki/config.yaml -target=write -config.expand-env=true -print-config-stderr log-config-reverse-order"
    ports:
      - 3102:3100
      - 7946
      - 9095
    volumes:
      - ./loki-config.yaml:/etc/loki/config.yaml
    healthcheck:
      test: [ "CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3100/ready || exit 1" ]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      <<: *loki-dns

  gateway:
    image: nginx:latest
    depends_on:
      - read
      - write
    entrypoint:
      - sh
      - -euc
      - |
        cat <<EOF > /etc/nginx/nginx.conf
        user  nginx;
        worker_processes  5;  ## Default: 1

        events {
          worker_connections   1000;
        }

        http {
          resolver 127.0.0.11;

          server {
            listen             3100;

            location = / {
              return 200 'OK';
              auth_basic off;
            }

            location = /api/prom/push {
              proxy_pass       http://write:3100\$$request_uri;
            }

            location = /api/prom/tail {
              proxy_pass       http://read:3100\$$request_uri;
              proxy_set_header Upgrade \$$http_upgrade;
              proxy_set_header Connection "upgrade";
            }

            location ~ /api/prom/.* {
              proxy_pass       http://read:3100\$$request_uri;
            }

            location = /loki/api/v1/push {
              proxy_pass       http://write:3100\$$request_uri;
            }

            location = /loki/api/v1/tail {
              proxy_pass       http://read:3100\$$request_uri;
              proxy_set_header Upgrade \$$http_upgrade;
              proxy_set_header Connection "upgrade";
            }

            location ~ /loki/api/.* {
              proxy_pass       http://read:3100\$$request_uri;
            }
          }
        }
        EOF
        /docker-entrypoint.sh nginx -g "daemon off;"
    ports:
      - "3100:3100"
    healthcheck:
      test: ["CMD", "service", "nginx", "status"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - loki

Loki配置文件（loki-config.yaml） #

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100
compactor:
  working_directory: /loki/boltdb-shipper-compactor
  shared_store: aws
schema_config:
  configs:
    - from: "2023-03-20"
      index:
        period: 24h
        prefix: index_
      object_store: s3
      schema: v12
      store: boltdb-shipper
storage_config:
  aws:
    bucketnames: lokidoki
    endpoint: http://s3.minio.local:9000
    insecure: true
    sse_encryption: false
    s3: s3://key:secret_key@http://s3.minio.local.:9000/lokidoki
    region: null
    s3forcepathstyle: true
    access_key_id: key
    secret_access_key: secret_key
    http_config:
      idle_conn_timeout: 5m
      response_header_timeout: 0s
      insecure_skip_verify: true
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/cache
    shared_store_key_prefix: loki-index/
    resync_interval: 5s
    shared_store: s3
  hedging:
    at: 250ms
    max_per_second: 20
    up_to: 3

ingester:
  lifecycler:
    address: 0.0.0.0
    join_after: "60s"
    observe_period: "5s"
    ring:
      replication_factor: 1
      kvstore:
        store: inmemory
    final_sleep: "0s"
  max_chunk_age: "240h"

ruler:
  storage:
    s3:
      bucketnames: lokidoki

# If you would like to disable reporting, uncomment the following lines:
#analytics:
#  reporting_enabled: false

排查与修复步骤 #

1. 清理S3配置冗余项 #

当前storage_config.aws中同时存在s3 URL和单独的密钥/端点配置，且URL中存在多余的点（s3.minio.local.），会导致配置冲突。修改后：

storage_config:
  aws:
    bucketnames: lokidoki
    endpoint: http://s3.minio.local:9000
    insecure: true
    sse_encryption: false
    region: "" # 用空字符串替代null，避免解析问题
    s3forcepathstyle: true
    access_key_id: key
    secret_access_key: secret_key
    http_config:
      idle_conn_timeout: 5m
      response_header_timeout: 0s
      insecure_skip_verify: true

删除冗余的s3字段，确保配置参数唯一且正确。

2. 指定Ingester的Chunk存储到S3 #

默认情况下Ingester可能将chunk存在本地临时目录，需明确指定存储后端：

ingester:
  lifecycler:
    address: 0.0.0.0
    join_after: "60s"
    observe_period: "5s"
    ring:
      replication_factor: 1
      kvstore:
        store: inmemory
    final_sleep: "0s"
  max_chunk_age: "240h"
  chunk_store_config:
    max_look_back_period: 0s
    store: s3 # 强制chunk存储到S3

3. 统一共享存储配置标识 #

compactor.shared_store设为aws，与boltdb_shipper.shared_store的s3不一致，统一为s3：

compactor:
  working_directory: /loki/boltdb-shipper-compactor
  shared_store: s3 # 修改为s3，保持与其他配置一致

4. 验证容器与MinIO的网络连通性 #

进入Loki的write容器，测试MinIO访问：

docker exec -it $(docker ps -q -f name=loki_write) curl http://s3.minio.local:9000

如果无法访问，需确保MinIO所在容器加入了loki网络，或调整DNS解析配置。

5. 推送测试日志触发存储操作 #

用curl推送一条测试日志，强制Loki生成chunk和index：

curl -H "Content-Type: application/json" -X POST http://localhost:3100/loki/api/v1/push --data '{"streams":[{"stream":{"job":"test"},"values":[["'$(date +%s000)'","test log message"]]}]}'

推送后等待3-5分钟（Loki的chunk刷新周期），再查看MinIO桶内容。

6. 检查容器内目录权限 #

Loki配置的/loki/index、/loki/cache目录需确保容器有读写权限：

docker exec -it $(docker ps -q -f name=loki_read) ls -ld /loki

如果权限不足，可在docker-compose中添加volume挂载，或启动容器时指定用户权限。

内容的提问来源于stack exchange，提问作者Serge