Prometheus目标配置、指标暴露及负载影响相关技术咨询

Hey there! Let's break down your questions one by one to get you sorted with Prometheus and your service setup.

1. What metrics does Prometheus scrape from itself? #

When Prometheus scrapes its own /metrics endpoint, it exposes a set of internal metrics that let you monitor its own health and performance. Key examples include:

• prometheus_target_scrapes_total: Total number of scrape attempts (successful + failed)
• prometheus_target_scrapes_success_total: Count of successful scrapes to targets
• prometheus_http_requests_total: Total HTTP requests made to Prometheus endpoints
• prometheus_tsdb_head_samples_appended_total: Number of data samples added to its time-series database
• process_cpu_seconds_total: CPU usage of the Prometheus process
• process_resident_memory_bytes: Memory footprint of the Prometheus process

These metrics give you full visibility into how the Prometheus server itself is running.

2. Can you expose a custom counter from Service B to Prometheus? #

Absolutely! This is a core use case for extending Prometheus to monitor application-specific behavior. You just need to:

• Add an HTTP endpoint (typically /metrics) to Service B that serves your custom metrics in Prometheus's standard text format.
• Configure Prometheus to scrape this endpoint just like it scrapes its own.

3. What should the endpoint return? #

Your endpoint needs to return plain text in the Prometheus exposition format. For your counter tracking successful queries to Service A, a valid response would look like this:

# HELP service_a_successful_queries_total Total number of successful queries from Service B to Service A
# TYPE service_a_successful_queries_total counter
service_a_successful_queries_total 42

Let's break this down:

• The # HELP line is optional but recommended—it explains what the metric measures.
• The # TYPE line is required—it tells Prometheus the metric type. Since you're incrementing on each success, counter is the right choice (counters only increase, unless the service restarts).
• The final line is the metric name followed by its current value.

If you have multiple instances of Service B, you can add labels to distinguish them:

service_a_successful_queries_total{instance="service-b-1", environment="production"} 42
service_a_successful_queries_total{instance="service-b-2", environment="production"} 38

4. Does Prometheus add extra load to the target? #

Yes, but it's usually negligible if configured properly. The load comes from two main sources:

• HTTP requests sent by Prometheus to your target's /metrics endpoint.
• The target generating and formatting the metrics response each time it's scraped.

To minimize impact:

• Set a reasonable scrape interval (default is 15s; if your counter doesn't need that granularity, bump it to 30s or 1 minute).
• Avoid generating expensive metrics on-the-fly—cache computed values if possible.
• Only expose metrics you actually need (don't bloat the response with unnecessary data).

In most cases, this load won't affect your service's performance, but it's good to keep these best practices in mind.

5. What configuration/data does a Prometheus target need? #

For a service to be a Prometheus target, here's what you need:

• An HTTP endpoint (typically /metrics) that returns metrics in the Prometheus exposition format.
• The endpoint must be reachable from the Prometheus server (no network/firewall blocks).
• Optional: Metadata labels (either embedded in your service's metrics, like the instance label above, or added via Prometheus's scrape configuration).

On the Prometheus side, you'll add a scrape job to your prometheus.yml config pointing to your Service B instances. Example:

scrape_configs:
  - job_name: 'service-b'
    static_configs:
      - targets: ['service-b-1:8080', 'service-b-2:8080']
    scrape_interval: 30s  # Adjust based on your monitoring needs

Once set up, you can build alerts (e.g., alert if the counter stops increasing for 5 minutes) or dashboards to monitor Service A's availability via Service B's query success rate.

内容的提问来源于stack exchange，提问作者Jwan622