Chainlink v1.0.1节点Prometheus监控缺失指标问题排查求助

Great question—let’s break down the most likely reasons you’re missing those specific metrics and walk through actionable fixes for each scenario:

1. Your Chainlink Node Isn’t Configured to Collect Detailed Metrics #

Many of the missing metrics (like pipeline_run_errors, adapter_perform_complete_total, and head tracker details) depend on your node having specific settings enabled to collect them. Here’s what to check:

• Enable Pipeline-related metrics: If your node isn’t running any Pipeline jobs, or hasn’t had Pipeline failures, these metrics won’t exist. First, confirm you have active Pipeline jobs configured. Additionally, verify your node’s environment variables include METRICS_ENABLED=true (required for all metrics) and METRICS_INCLUDE_DETAILED=true (a flag that exposes more granular metrics in Chainlink v1.0.1).
• TxPool metrics require Ethereum client setup: Metrics like txpool_local, txpool_queued, and txpool_pending rely on your connected Ethereum client (e.g., Geth, Besu) exposing its TxPool API. Make sure your ETH client is started with appropriate flags (like --txpool.api for Geth) and that your Chainlink node’s ETH_URL points to this fully enabled endpoint. Some versions also require ENABLE_TXPOOL_METRICS=true as an environment variable for the Chainlink node.

2. Prometheus Isn’t Authenticating to the Chainlink Node’s Metrics Endpoint #

Your current Prometheus config doesn’t include basic auth, but Chainlink nodes secure the /metrics endpoint with the same API_USER and API_PASSWORD you set for the node’s API. Without authentication, the node will only return a subset of public metrics, blocking access to more internal metrics like pipeline errors and txpool data.

Update your Prometheus job config to include basic auth:

job_name: 'chainlink_p1'
static_configs:
  targets: ['CONTAINER-ID:6688']
metrics_path: '/metrics'
scheme: https
tls_config:
  insecure_skip_verify: true
basic_auth:
  username: "<YOUR_CHAINLINK_API_USER>"
  password: "<YOUR_CHAINLINK_API_PASSWORD>"

After updating, restart Prometheus and re-scrape the metrics endpoint.

3. Some Metrics Don’t Exist in Chainlink v1.0.1 #

Chainlink’s metrics set expanded significantly in later versions. A handful of the metrics you’re missing weren’t introduced until after v1.0.1, including:

• adapter_perform_complete_total
• run_status_update_total
• head_tracker_callback_execution_duration

While you mentioned testing other versions, double-check that you tried a stable release like v1.1.0 or newer—these versions should include the missing metrics. If you’re locked into v1.0.1 for compatibility reasons, you’ll need to either accept these metrics aren’t available or build a custom exporter to scrape the node’s API for equivalent data.

4. Metrics Only Appear When Triggered by Node Activity #

Some of the missing metrics are event-driven—they won’t show up until the node performs a specific action. For example:

• pipeline_run_errors: Only appears when a Pipeline job fails. Try creating a test Pipeline that’s guaranteed to fail (e.g., a job that calls a non-existent adapter) to trigger this metric.
• job_subscriber_subscriptions: Only exists if your node is subscribed to active jobs like Flux Monitors or OCR feeds. Confirm you have active jobs running on the node.
• tx_manager_num_tx_reverted: Only populates when a transaction gets reverted on-chain. Send a test transaction that will fail (e.g., insufficient gas) to generate this metric.

After triggering these actions, wait a few minutes for Prometheus to scrape the endpoint again and check if the metrics appear.

5. Verify Connectivity and Node Logs for Errors #

Even though you’re getting some metrics, there might be partial connectivity issues or node-side errors preventing full metric collection:

• Test manual metric scraping: From inside your Prometheus container, run curl -k -u <API_USER>:<API_PASS> https://CONTAINER-ID:6688/metrics and inspect the output. If you still don’t see the missing metrics here, the problem is with the Chainlink node, not Prometheus.
• Check Chainlink node logs: Look for errors related to metric registration (e.g., failed to register metric) or disabled features. Logs might reveal that certain modules (like Pipeline or TxPool monitoring) aren’t loading correctly.

内容的提问来源于stack exchange，提问作者Throbac Labs