Grafana Loki分布式部署报错：Bigtable请求列族未找到

开发者阿泽

2026-8-6

问题：Loki Ingester无法向Bigtable推送索引（列族未找到错误）

我在GKE集群部署了使用GCS和Bigtable的Loki分布式Helm Chart，检查loki ingester Pod时遇到如下错误：

level=error ts=2023-01-06T13:01:59.107018356Z caller=flush.go:146 org_id=fake msg="failed to flush user" err="store put chunk: rpc error: code = NotFound desc = Error while mutating the row '08:fake:d19362:logs' (projects//instances/mybigtable/tables/loki_index_2766) : Requested column family not found.

我已按照升级文档准备了ConfigMap（配置见下方），且Loki分布式服务账户已通过IAM服务账户注解配置工作负载身份权限，但日志索引仍无法推送到Bigtable，怀疑需要调整table_manager配置，但不确定具体方案，求建议。

ConfigMap配置

---
# Source: loki-distributed/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: loki-loki-distributed
  labels:
    helm.sh/chart: loki-distributed-0.67.0
    app.kubernetes.io/name: loki-distributed
    app.kubernetes.io/instance: loki
    app.kubernetes.io/version: "2.6.1"
    app.kubernetes.io/managed-by: Helm
data:
  config.yaml: |
    auth_enabled: false
    chunk_store_config: 
      max_look_back_period: 672h
    compactor:
      retention_enabled: true
      shared_store: filesystem
    distributor:
      ring:
        kvstore:
          store: memberlist
    frontend:
      compress_responses: true
      log_queries_longer_than: 5s
      tail_proxy_url: http://loki-loki-distributed-querier:3100
    frontend_worker:
      frontend_address: loki-loki-distributed-query-frontend:9095
    ingester:
      chunk_block_size: 262144
      chunk_encoding: snappy
      chunk_idle_period: 30m
      chunk_retain_period: 1m
      lifecycler:
        ring:
          kvstore:
            store: memberlist
          replication_factor: 1
      max_transfer_retries: 0
      wal:
        dir: /var/loki/wal
    limits_config:
      enforce_metric_name: false
      retention_period: 30d
      max_cache_freshness_per_query: 10m
      reject_old_samples: true
      reject_old_samples_max_age: 168h
      split_queries_by_interval: 15m
    memberlist:
      join_members:
      - loki-loki-distributed-memberlist
    query_range:
      align_queries_with_step: true
      cache_results: true
      max_retries: 5
      results_cache:
        cache:
          enable_fifocache: true
          fifocache:
            max_size_items: 1024
            ttl: 24h
    ruler:
      alertmanager_url: https://alertmanager.xx
      external_url: https://alertmanager.xx
      ring:
        kvstore:
          store: memberlist
      rule_path: /tmp/loki/scratch
      storage:
        local:
          directory: /etc/loki/rules
        type: local
    runtime_config:
      file: /var/loki-distributed-runtime/runtime.yaml
    schema_config:
      configs:
      - from: 2023-01-01
        store: bigtable
        object_store: gcs
        schema: v12
        index:
          prefix: loki_index_
          period: 168h

    server:
      http_listen_port: 3100
    storage_config:
      gcs:
        bucket_name: gcs-logs
      bigtable:
        instance: loki-indexes
        project: myproject-id
      boltdb_shipper:        
        active_index_directory: /loki/boltdb-shipper-active
        cache_location: /loki/boltdb-shipper-cache
        cache_ttl: 24h
        shared_store: gcs
        
        #active_index_directory: /var/loki/index
        #cache_location: /var/loki/cache
        #cache_ttl: 168h
        #shared_store: filesystem
      
      filesystem:
        directory: /var/loki/chunks
    table_manager:
      retention_deletes_enabled: true
      retention_period: 672h

服务账户配置

---
# Source: loki-distributed/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: loki-loki-distributed
  labels:
    helm.sh/chart: loki-distributed-0.67.0
    app.kubernetes.io/name: loki-distributed
    app.kubernetes.io/instance: loki
    app.kubernetes.io/version: "2.6.1"
    app.kubernetes.io/managed-by: Helm
  annotations:
    iam.gke.io/gcp-service-account: <iam-sa-account>@<project-id>.iam.gserviceaccount.com
automountServiceAccountToken: true

解决方案建议

1. 确认Bigtable表的列族存在

错误提示明确是Requested column family not found，Loki使用Bigtable时默认需要名为f的列族：

登录GCP控制台，找到对应的Bigtable实例loki-indexes和表loki_index_2766
检查是否存在名为f的列族，若缺失则手动创建

2. 调整table_manager配置以自动管理列族

当前table_manager配置缺少Bigtable相关设置，需添加配置块让Loki自动维护表结构：

table_manager:
  retention_deletes_enabled: true
  retention_period: 672h
  bigtable:
    create_tables: true  # 允许Loki自动创建表
    column_family: "f"   # 显式指定列族名称，与默认值保持一致

3. 验证Bigtable实例和项目ID匹配

日志中表路径显示projects//instances/mybigtable/tables/loki_index_2766，存在项目名称为空、实例名与配置不符的问题：

确认storage_config.bigtable.project的myproject-id是正确的GCP项目ID
检查日志中的实例名mybigtable是否与配置的loki-indexes一致，若不一致需重新部署ConfigMap并重启Pod

4. 验证IAM权限

确保工作负载身份绑定的IAM服务账户拥有以下权限：

bigtable.tables.create（创建表和列族）
bigtable.tables.update（修改表结构）
bigtable.tables.data.readwrite（读写数据）

5. 重启相关Pod

修改ConfigMap后，重启ingester和table-manager Pod让配置生效：

kubectl rollout restart deployment loki-loki-distributed-ingester
kubectl rollout restart deployment loki-loki-distributed-table-manager

内容的提问来源于stack exchange，提问作者ARINDAM BANERJEE