排查Stack Driver Monitoring高额调用费用：寻找google.monitoring.v3.MetricService.ListTimeSeries请求触发源

Hey there! Let's walk through practical steps to figure out where those 1M monthly ListTimeSeries requests are coming from—especially since you're seeing unexpected costs in a development environment with no live traffic.

Step 1: Use Cloud Audit Logs to Trace API Calls #

Cloud Audit Logs records all calls to GCP APIs, including Monitoring. Here's how to leverage it:

• Open the Cloud Logging page in your GCP Console.
• Use this query to filter specifically for your target API calls:

  protoPayload.methodName="google.monitoring.v3.MetricService.ListTimeSeries"
  

• For each log entry, check these key fields to pinpoint the source:
  • protoPayload.authenticationInfo.principalEmail: This shows the service account or user account making the request.
  • protoPayload.requestMetadata.callerIp: An external IP might point to a local dev machine; a GCP internal IP likely means a managed GCP service is the culprit.
  • protoPayload.resourceName: Reveals which metrics/resources the request targets (e.g., a specific VM, Cloud Function, or custom metric), giving you clues about what's triggering it.

Step 2: Check for Automated GCP Service Calls #

Some GCP services automatically pull monitoring metrics in the background, even with no app traffic:

• Cloud Console Dashboards: If your team frequently checks monitoring dashboards, the console itself makes periodic ListTimeSeries calls to refresh metrics.
• Alerting Policies: Active alert rules poll metrics via this API to evaluate conditions—even if no alerts are firing, the polling still happens.
• Managed Services: App Engine, Cloud Run, or GKE have built-in monitoring pipelines that might trigger these requests to track infrastructure health.

Step 3: Audit Service Account Permissions and Usage #

Head to the IAM & Admin > Service Accounts page in GCP Console:

• List all service accounts with Monitoring Viewer or Monitoring Editor permissions (these are the roles required to call ListTimeSeries).
• For each suspicious account, check its Activity tab to see if it's been making these API calls.
• Also, look for service accounts linked to your own applications—if you've integrated the Cloud Monitoring SDK (e.g., in Python, Java, or Go code), your app might be configured to pull metrics on a schedule without you realizing it.

Step 4: Trace API Calls from Your Applications #

If you suspect the requests are coming from your code:

• Enable Cloud Trace for your apps. Trace will capture full API call traces, including calls to the Monitoring API, and show you the exact code path triggering the request.
• Use Cloud Debugger to set breakpoints in code that interacts with Monitoring, verifying if it's executing unexpectedly (e.g., on a cron job that's still running in dev).

Step 5: Test with Temporary Permission Restrictions #

To confirm a suspected source:

• Temporarily remove Monitoring Viewer permissions from a service account or user you think is responsible.
• Wait a few hours, then check the ListTimeSeries request volume in Cloud Monitoring's API Metrics dashboard. If the volume drops, you've found your culprit.

Bonus: Reduce Costs Without Disabling the API #

Since disabling the Cloud Monitoring API breaks dependent services (like alerting and dashboards), try these cost-saving tweaks instead:

• Adjust the sampling frequency for custom metrics or dashboard widgets to pull less often.
• Delete unused alerting policies or dashboards that might be polling metrics unnecessarily.
• Disable non-essential default metrics that aren't critical for your development workflow.

内容的提问来源于stack exchange，提问作者Ayyappa