Grafana仅展示近3天指标而非全月，Thanos-Prometheus栈配置排查

Let's break down why your 30-day node-exporter uptime query is only pulling the last 3 days of data, even though your S3 bucket has metrics going back to mid-September. Here are the most likely culprits and actionable fixes:

1. Verify Thanos Sidecar is Actually Uploading Blocks to S3 #

"指标推送无报错" doesn't guarantee blocks are successfully landing in S3. Let's confirm:

• Navigate directly to your S3 bucket and check for timestamp-named block directories (e.g., 1693545600). Each valid block should contain a meta.json file.
• Double-check your Sidecar startup flags:
  • Ensure --objstore.config-file points to the correct S3 config (matching bucket name, region, and credentials).
  • Confirm --prometheus.url=http://localhost:9090 (adjust port if your Prometheus uses a non-default one) is present—this lets the Sidecar pull blocks from Prometheus.
• Check Sidecar logs for upload confirmation messages like uploaded block <block-id> to object storage. If you don't see these, there's a silent failure in the upload process.

2. Confirm Thanos Store Can Access All Historical Blocks #

You mentioned the Store "can正常读取S3桶", but let's validate it's actually seeing the old blocks:

• On the Store machine, run this command with your S3 config:

  thanos store ls --objstore.config-file=/path/to/your/s3-config.yaml
  

  This should list all blocks in S3, including the mid-September ones. If old blocks are missing here:
  • Either the Store's S3 config is incorrect (double-check permissions and bucket name), or
  • The blocks uploaded by the Sidecar have incorrect permissions (ensure the Store has read access to all objects in the bucket).
• Make sure the Store's startup flags include the correct --objstore.config-file and that the process has sufficient IAM permissions to read from S3.

3. Check Thanos Compactor's Functionality #

Your Compactor is configured with --retention.resolution-raw=3d, which deletes raw Prometheus blocks after 3 days—but only after compressing them into 5m and 1h resolution blocks. If the Compactor isn't working, those raw blocks get deleted by Prometheus, and no aggregated blocks exist in S3 for older data.

• First, check Compactor logs for compaction success messages like compacted block <block-id> into <new-block-id>. If you don't see these, the Compactor isn't running properly.
• Verify the Compactor's startup flags:
  • Ensure --objstore.config-file matches the S3 config used by the Sidecar/Store.
  • Confirm no other Compactor instances are running—Compactors require exclusive access to the S3 bucket's compaction directory.
• Check if the Compactor has write permissions to the S3 bucket—without it, can't create aggregated blocks.

4. Validate Thanos Query <-> Store Connection #

Even if you configured the Query to point to the Store, gRPC connectivity issues might prevent the Query from pulling historical data:

• On the Query machine, run:

  thanos query stores --endpoint=localhost:29090
  

  This should list your Store instance with a status of UP. If the Store is missing or marked DOWN:
  • Check if the Store's --grpc-address flag is set (default is 0.0.0.0:10901) and the port is open in firewalls between the Query and Store machines.
  • Ensure the Query's startup flags include --store=<store-ip>:10901 (replace with your Store's actual IP/port).

5. Ensure Prometheus Blocks Are Uploaded Before Deletion #

Prometheus deletes blocks after 3 days, so the Sidecar needs to upload them before that happens:

• Confirm the Sidecar's --tsdb.path flag points to your Prometheus TSDB directory (e.g., /var/lib/prometheus)—this lets the Sidecar monitor new blocks and upload them immediately.
• Compare Prometheus logs (look for deleting block <block-id>) with Sidecar upload logs. If blocks are deleted before being uploaded, adjust Prometheus's retention or fix the Sidecar's upload latency.

内容的提问来源于stack exchange，提问作者strzelecki.maciek