Loki规则目录读取失败：无法找到/tmp/loki/rules/fake目录

环境信息 #

• Loki版本：{"version":"2.6.1","revision":"6bd05c9a4","branch":"HEAD","buildUser":"root@ea1e89b8da02","buildDate":"2022-07-18T08:49:07Z","goVersion":""}
• Grafana版本：9.0.5（截图显示）

当前配置 #

Loki Helm Chart配置 #

loki:
  enabled: true

  image:
    repository: grafana/loki
    pullPolicy: Always
    pullSecrets:
      - registry
  priorityClassName: normal
  resources:
    limits:
      memory: 3Gi
      cpu: 0
    requests:
      memory: 0
      cpu: 0
  config:
    chunk_store_config:
      max_look_back_period: 30d
    table_manager:
      retention_deletes_enabled: true
      retention_period: 30d
    query_range:
      split_queries_by_interval: 0
      parallelise_shardable_queries: false
    querier:
      max_concurrent: 2048
    frontend:
      max_outstanding_per_tenant: 4096
      compress_responses: true
    ingester:
      wal:
        enabled: true
        dir: /tmp/wal
    schema_config:
      configs:
        - from: 2022-12-05
          store: boltdb-shipper
          object_store: filesystem
          schema: v11
          index:
            prefix: index_
            period: 24h
    storage_config:
      boltdb_shipper:
        active_index_directory: /tmp/loki/boltdb-shipper-active
        cache_location: /tmp/loki/boltdb-shipper-cache
        cache_ttl: 24h
        shared_store: filesystem
      filesystem:
        directory: /tmp/loki/chunks
    compactor:
      working_directory: /tmp/loki/boltdb-shipper-compactor
      shared_store: filesystem
    ruler:
      storage:
        type: local
        local:
          directory: /tmp/loki/rules/
      ring:
        kvstore:
          store: inmemory
      rule_path: /tmp/loki/rules-temp
      alertmanager_url: http://onprem-kube-prometheus-alertmanager.svc.mylocal-monitoring:9093
      enable_api: true
      enable_alertmanager_v2: true
  write:
    extraVolumeMounts:
      - name: rules-config
        mountPath: /tmp/loki/rules/fake/
    extraVolumes:
      - name: rules-config
        configMap:
          name: rules-cfgmap
          items:
            - key: "rules.yaml"
              path: "rules.yaml"
  read:
    extraVolumeMounts:
      - name: rules-config
        mountPath: /tmp/loki/rules/fake/
    extraVolumes:
      - name: rules-config
        configMap:
          name: rules-cfgmap
          items:
            - key: "rules.yaml"
              path: "rules.yaml"

promtail:
  image:
    registry: docker
    pullPolicy: Always
  imagePullSecrets:
    - name: registry
  priorityClassName: normal
  resources:
    limits:
      memory: 256Mi
      cpu: 0
    requests:
      memory: 0
      cpu: 0
  livenessProbe:
    failureThreshold: 5
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 10
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 1
  config:
    snippets:
      pipelineStages:
        - cri: {}
      common:
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_node_name
          target_label: node_name
        - action: replace
          source_labels:
            - __meta_kubernetes_namespace
          target_label: namespace
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_container_name
          target_label: container
        - action: replace
          replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels:
            - __meta_kubernetes_pod_uid
            - __meta_kubernetes_pod_container_name
          target_label: __path__
        - action: replace
          replacement: /var/log/pods/*$1/*.log
          regex: true/(.*)
          separator: /
          source_labels:
            - __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
            - __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
            - __meta_kubernetes_pod_container_name
          target_label: __path__

monitoring:
  enabled: false

networkPolicies:
  enabled: false

规则ConfigMap配置 #

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rules-cfgmap
  namespace: mylocal-monitoring
data:
  rules.yaml: |
    groups:
      - name: PrometheusAlertsGroup
    rules:
      - alert: test1
      expr: |
        1 > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: TEST: testing test
          description: test

遇到的问题 #

执行curl -X GET localhost:3100/loki/api/v1/rules检查规则时，返回错误：

unable to read rule dir /tmp/loki/rules/fake: open /tmp/loki/rules/fake: no such file or directory

手动创建/tmp/loki/rules/fake/rules.yaml后可正常工作，但需要实现自动挂载，无需手动操作。

已尝试的修改 #

调整write和read节点的挂载配置，直接将ConfigMap挂载到文件路径：

write:
  extraVolumeMounts:
    - name: rules-conf
      mountPath: /tmp/loki/rules/fake/rules.yaml
  extraVolumes:
    - name: rules-conf
read:
  extraVolumeMounts:
    - name: rules-conf
      mountPath: /tmp/loki/rules/fake/rules.yaml
  extraVolumes:
    - name: rules-conf

问题依旧存在。

排查解决思路 #

1. 提前创建规则目录
   Kubernetes挂载ConfigMap到目录时，不会自动创建父目录。添加initContainer在Loki启动前创建目标目录：

   loki:
     write:
       initContainers:
         - name: create-rule-dir
           image: busybox:latest
           command: ["mkdir", "-p", "/tmp/loki/rules/fake"]
           volumeMounts:
             - name: rules-config
               mountPath: /tmp/loki/rules/fake
     read:
       initContainers:
         - name: create-rule-dir
           image: busybox:latest
           command: ["mkdir", "-p", "/tmp/loki/rules/fake"]
           volumeMounts:
             - name: rules-config
               mountPath: /tmp/loki/rules/fake
   

   若Locker使用非root用户运行，需添加chown命令确保目录权限正确：command: ["sh", "-c", "mkdir -p /tmp/loki/rules/fake && chown loki:loki /tmp/loki/rules/fake"]

2. 修正规则文件YAML格式
   当前规则文件存在缩进错误，rules应属于groups下的子项，正确格式：

   groups:
     - name: PrometheusAlertsGroup
       rules:
         - alert: test1
           expr: 1 > 0
           for: 0m
           labels:
             severity: critical
           annotations:
             summary: TEST: testing test
             description: test
   

   格式错误会导致Loki无法解析规则，即使文件挂载成功也不生效。

3. 验证Pod挂载状态
   执行命令检查目录和文件是否存在：

   kubectl exec -n mylocal-monitoring <loki-pod-name> -- ls -l /tmp/loki/rules/fake
   

   • 若目录不存在：说明initContainer未配置或执行失败
   • 若目录存在但无文件：说明ConfigMap挂载配置有误，检查extraVolumes中的ConfigMap名称、key是否匹配

4. 检查Helm渲染后的实际配置
   查看Helm渲染后的Pod YAML，确认挂载配置是否正确生效：

   helm get manifest -n mylocal-monitoring <loki-release-name>
   

内容的提问来源于stack exchange，提问作者Brian Brown