如何为租户性能监控告警添加请求量阈值的附加触发条件？

Great question! Since most out-of-the-box monitoring solutions don’t natively support this kind of multi-condition alerting tied to per-tenant metrics, we’ll cover both straightforward implementations and some creative workarounds tailored to your use case.

1. Straightforward Composite Alert Rules (Using Built-in Monitoring Features) #

Most modern monitoring tools (like Prometheus+Alertmanager, Datadog, New Relic) let you build composite alert rules that combine multiple metric conditions with logical ANDs. Here’s how to structure this:

Example with Prometheus/PromQL #

If you’re using Prometheus, you can write a single PromQL query that checks both conditions simultaneously, grouped by tenant:

# Check 24h p50 latency > 100ms AND 24h total requests > 100
histogram_quantile(0.5, sum by (tenant, le) (rate(request_duration_bucket[24h]))) > 0.1
AND
sum by (tenant) (rate(request_count[24h])) * 86400 > 100

• The first part calculates the 24-hour rolling p50 latency (converted to seconds, hence >0.1 for 100ms).
• The second part calculates the total 24-hour requests by multiplying the per-second rate by 86400 seconds in a day.
• This query will only return results for tenants that meet both conditions, which you can feed directly into an alert rule.

For SaaS Monitoring Tools (Datadog, New Relic) #

Use the tool’s custom alert builder to:

• Add two separate metric conditions (one for latency p50, one for total requests).
• Set the alert to trigger only when both conditions are true (look for "AND" logic in the rule configuration).
• Ensure both conditions are filtered and grouped by your tenant identifier.

2. Creative Pre-Aggregation Workaround #

If your monitoring tool doesn’t support complex composite rules well, you can pre-aggregate the metrics to simplify the alert logic:

• Build a lightweight data pipeline (using tools like Flink, Spark Streaming, or even a cron job with a script) that runs every 24 hours.
• For each tenant, calculate two values: total requests in the last 24h, and p50 latency in the last 24h.
• Write these pre-computed values to your time-series database as new metrics (e.g., tenant_daily_requests_total, tenant_daily_p50_latency).
• Now your alert rule becomes simple: trigger when tenant_daily_requests_total > 100 AND tenant_daily_p50_latency > 100ms.

This approach is especially useful if you need to add more complex conditions later (like excluding maintenance windows for specific tenants).

3. Adaptive Thresholds (Bonus Creative Twist) #

While your requirement specifies a fixed 100-request threshold, you could extend this with adaptive thresholds to avoid false positives for tenants with normally very low traffic:

• Calculate a baseline request volume for each tenant (e.g., the 90th percentile of their daily request counts over the last 30 days).
• Set the threshold as a multiple of this baseline (e.g., baseline * 2 or max(100, baseline * 1.5)).
• This ensures you only alert when latency is high and traffic is unusual for that specific tenant, not just above an arbitrary fixed number.

Testing the Rule #

Don’t forget to validate your alert logic with test scenarios:

• Simulate a tenant with p50 >100ms but <100 requests: no alert should trigger.
• Simulate a tenant with >100 requests but p50 <100ms: no alert should trigger.
• Simulate a tenant meeting both conditions: alert should trigger as expected.

内容的提问来源于stack exchange，提问作者Dave New