Thanos for Prometheus无法向S3桶存储数据求助

我正在搭建监控架构，为解决Prometheus存储容量过大问题，计划通过Thanos将数据存储到S3桶，所有组件已容器化。目前所有Thanos容器状态正常，日志显示同步成功，但数据并未写入S3桶，已确认S3用户拥有目标桶的读写、列出及删除权限，现将配置及日志信息列出，请求排查建议。

Docker Compose配置 #

version: '3.9'

services:

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: on-failure:5
    network_mode: host
    volumes:
      - ./prometheus/:/etc/prometheus/
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=1d"
    user: "0"

  thanos-sidecar:
    image: thanosio/thanos:main-2024-02-12-f28680c
    container_name: thanos-sidecar
    restart: unless-stopped
    depends_on: 
      - prometheus
    extra_hosts:
      - "host.docker.internal:host-gateway"
    command:
      - "sidecar"
      - "--http-address=0.0.0.0:19090"
      - "--grpc-address=0.0.0.0:19190"
      - "--prometheus.url=http://host.docker.internal:9090"

  thanos-store:
    image: thanosio/thanos:main-2024-02-12-f28680c
    container_name: thanos-store
    restart: unless-stopped
    depends_on: 
      - prometheus
    expose:
      - "10901"
    command:
      - "store"
      - "--grpc-address=0.0.0.0:10901"
      - "--data-dir=/thanos-data"
      - "--objstore.config-file=/etc/thanos/s3.yaml"
      - "--sync-block-duration=0.5m"
    volumes:
      - ./thanos:/etc/thanos
      - ./thanos/thanos-data:/thanos-data

  thanos-query:
    image: thanosio/thanos:main-2024-02-12-f28680c
    container_name: thanos-query
    restart: unless-stopped
    ports: 
      - "10902:10902"
    expose: 
      - "10902"
    depends_on: 
      - prometheus
    command:
      - "query"
      - "--http-address=0.0.0.0:10902"
      - "--grpc-address=0.0.0.0:10901"
      - "--store=thanos-store:10901"

S3 配置文件（s3.yaml） #

type: S3
config:
  bucket: prometheus-storaging
  endpoint: s3.eu-west-1.amazonaws.com 
  access_key: <user-access-key>
  secret_key: <user-secret-key>
  insecure: false
  region: eu-west-1

Thanos Store组件同步日志 #

ts=2024-02-12T14:58:47.108623799Z caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=26.465216ms duration_ms=26 cached=0 returned=0 partial=0
ts=2024-02-12T14:59:17.110948605Z caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=28.802605ms duration_ms=28 cached=0 returned=0 partial=0
ts=2024-02-12T14:59:47.109934224Z caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=27.379893ms duration_ms=27 cached=0 returned=0 partial=0
ts=2024-02-12T15:00:17.113818181Z caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=31.517016ms duration_ms=31 cached=0 returned=0 partial=0
ts=2024-02-12T15:00:47.11444593Z caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=31.438836ms duration_ms=31 cached=0 returned=0 partial=0
ts=2024-02-12T15:01:17.113290234Z caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=31.118912ms duration_ms=31 cached=0 returned=0 partial=0
ts=2024-02-12T15:01:47.111825861Z caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=29.215559ms duration_ms=29 cached=0 returned=0 partial=0
ts=2024-02-12T15:02:17.112276432Z caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=29.431009ms duration_ms=29 cached=0 returned=0 partial=0

排查建议 #

• 补全Thanos Sidecar的对象存储配置：当前Sidecar仅连接Prometheus，未配置上传S3的参数，需添加--objstore.config-file=/etc/thanos/s3.yaml命令行参数，同时将s3.yaml所在目录挂载到Sidecar容器中，比如在Sidecar的volumes添加- ./thanos:/etc/thanos。
• 检查Prometheus TSDB目录的访问权限：当前Prometheus未挂载TSDB存储目录（默认路径为/prometheus），Sidecar无法读取生成的块文件。需给Prometheus添加volume配置- ./prometheus/data:/prometheus，确保Sidecar能通过网络或挂载路径访问这些块。
• 查看Thanos Sidecar日志：Store日志显示returned=0说明无新块可同步，大概率是Sidecar未上传块到S3。需检查Sidecar日志，确认是否有块上传尝试、是否存在权限或网络错误。
• 验证Prometheus块生成机制：Prometheus默认每2小时生成一个TSDB块，需等待足够时间让块生成，同时确认--storage.tsdb.retention.time=1d配置未提前清理块。
• 测试Sidecar到S3的连通性：进入Sidecar容器执行curl s3.eu-west-1.amazonaws.com，确认网络无阻塞，同时检查服务器安全组、防火墙是否允许出站访问S3端口。
• 切换Thanos稳定版本：当前使用的是main分支快照版本，可能存在稳定性问题，建议替换为官方稳定版本（如v0.33.0）重新测试。
• 修正Thanos Query的端口冲突：当前Query的--grpc-address设置为0.0.0.0:10901，与Store的端口冲突，需修改为不同端口（如0.0.0.0:10903），避免端口占用导致组件异常。

内容的提问来源于stack exchange，提问作者nico