Prometheus与Grafana负载场景下虚假指标峰值问题排查求助

First off, since you've confirmed the application isn't generating bad data, we can focus our investigation on Prometheus itself, the scrape pipeline, and potential edge cases with prom-client. Here's a structured list of checks to run:

1. Validate Prometheus Storage Layer Stability #

• Check Prometheus logs for disk-related errors (like out of disk space, IO timeout, or block write/fusion failures) during high load. False metrics often stem from corrupted writes when the storage layer can't keep up.
• Inspect disk health: Use df -h to check free space, and iostat to monitor IO wait times and throughput. If disk usage is near 100% or IO wait spikes, Prometheus can't reliably write/read data, leading to dirty or incorrect metrics.
• Review storage configuration: Even if you're using defaults, verify settings like storage.tsdb.retention.time (ensure it's not causing excessive disk bloat) and enable storage.tsdb.wal-compression if you haven't already. Uncompressed WAL logs can cripple IO performance under load, triggering write anomalies.

2. Tune Prometheus Query & Load Handling #

• Limit query concurrency: Prometheus has no strict default limits on concurrent queries. When Grafana floods it with large time-range requests, exhausted query threads can cause result corruption. Add startup flags like --query.max-concurrency=20 and --query.max-samples=50000000 to prevent single queries from hogging resources.
• Test your PromQL: While your query avg(rate(udp_uplink_receive_duration_seconds_bucket{ success="true"}[1h])) looks basic, rate on large time ranges processes massive sample sets. Try narrowing the time window or switching to irate temporarily—if the false metrics disappear, it could point to sample alignment issues under load.

3. Dig Into prom-client's Metric Exposure #

• Validate metric endpoint reliability: Even if the app generates correct data, high-frequency scrapes (your 5s interval is aggressive) might trigger race conditions in prom-client. Use curl to repeatedly hit your app's /metrics endpoint during high load, save the outputs, and compare for unexpected bucket values or missing samples. Node.js is single-threaded, but async metric updates could cause inconsistent data during concurrent scrapes.
• Update prom-client: Older versions have known bugs around concurrent metric serialization. Upgrade to the latest stable release to rule out fixed issues.

4. Check Network & Scrape Pipeline #

• Test network connectivity: Use ping or mtr to check for packet loss or latency spikes between Prometheus and your Node.js app during high load. Scrape timeouts or incomplete metric transfers (logged as context deadline exceeded in Prometheus) can lead to partial data being stored as false metrics.
• Verify label handling: Your honor_labels: true config is fine, but double-check that the app's metric labels don't have duplicates or invalid characters. High load could exacerbate parsing errors in Prometheus, creating broken time series.

5. Monitor Prometheus Process & Resource Limits #

• Track resource usage: Use top/htop to watch CPU and memory usage. If Prometheus hits 100% CPU or runs out of memory (OOM), it'll behave erratically—check system logs with dmesg | grep oom to see if the process was killed by the OOM killer.
• Container resource limits: If running Prometheus in a container, ensure CPU/memory quotas are sufficient. Resource throttling under load can cause data processing failures and false metrics.

内容的提问来源于stack exchange，提问作者Martin Hudec