基于Prometheus+Loki+Grafana+Promtail架构，如何配置Promtail丢弃含healthcheck/health-check的日志

Hey there! Let's get those unwanted health check logs filtered out of your Promtail setup. Since you're using the Helm chart, we'll adjust the values configuration to add a filtering pipeline that drops logs containing healthcheck or health-check. Here's a step-by-step guide:

Step 1: Access and Modify Your Promtail Helm Values #

Open up your custom Promtail values file (or reference the chart's default values as a base). Navigate to the config.scrape_configs section—this is where all your log scraping jobs are defined. For most Kubernetes environments, you'll focus on the kubernetes job (the default one for scraping cluster logs).

Step 2: Add the Drop Pipeline Stage #

Within your target scrape job (e.g., kubernetes), add a pipeline_stages block (if it doesn't already exist) and include a drop stage that matches the unwanted keywords. This stage will scan the log message field and discard any entry that contains either term.

Here's the exact configuration snippet to insert:

config:
  scrape_configs:
  - job_name: kubernetes
    # Keep all your existing configuration here (like kubernetes_sd_configs, relabel_configs, etc.)
    pipeline_stages:
    - drop:
        expression: "healthcheck|health-check"
        source: "message" # Adjust this if your logs store the content in a different field (e.g., "log" or "msg")

Step 3: Apply the Configuration Update #

Once you've saved your modified values file, upgrade your Promtail Helm release to apply the changes. Run this command (replace promtail with your release name and your-values.yaml with your file path):

helm upgrade promtail grafana/promtail -f your-values.yaml

Step 4: Verify the Filter Works #

After the upgrade completes, you can verify the filtering is working by:

• Checking the Promtail pod logs to confirm no errors in the pipeline configuration
• Querying Loki via Grafana to ensure logs containing healthcheck or health-check are no longer appearing in your dashboards

A quick note: If your logs store the main content in a field other than message (some applications use log or msg), make sure to update the source value in the drop stage to match that field name.

内容的提问来源于stack exchange，提问作者Gauri Joshi