Promtail采集目标失败排查求助：Kubernetes环境下无法推送日志至Loki

Let’s break down your issue step by step—based on the logs, metrics, and config you shared, here’s what’s going wrong and how to fix it:

1. Empty Labels Are Dropping Your Targets (Root Cause) #

The biggest red flag is your Promtail metric promtail_targets_failed_total{reason="empty_labels"} 2280. This means hundreds of potential log targets are being discarded because required labels are missing after relabeling.

Looking at your scrape configs, jobs like kubernetes-pods-app-kubernetes-io-name have a rule that drops any target where the app label is empty:

- action: drop
  regex: ''
  source_labels:
  - app

This rule triggers when your pods don’t have an app.kubernetes.io/name label (since that’s what populates the app label here), or if the label extraction fails for any reason.

Fixes:

• Add missing labels to pods: If your workloads don’t have app.kubernetes.io/name or app labels, update their deployment/statefulset manifests to include these standard Kubernetes labels.
• Adjust relabel rules: If you need to collect logs from all pods (even unlabeled ones), modify the rules to avoid dropping empty labels, or set a fallback value. For example, in the kubernetes-other job, add a rule to set app to the pod name if no other label exists:

  - action: replace
    source_labels: [__meta_kubernetes_pod_name]
    target_label: app
    replacement: $1
  

2. Promtail Isn’t Mounting Node Log Directories Correctly #

Your Promtail pod’s /var/logs directory is empty—this tells me the pod isn’t accessing the node’s log files. Promtail needs direct access to /var/log/pods (where Kubernetes stores pod logs) and /var/lib/docker/containers (where raw container logs live) on the host node.

Fix:
Check your Promtail DaemonSet volume mounts to ensure they include these host paths:

volumes:
  - name: varlog
    hostPath:
      path: /var/log
  - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers
      type: DirectoryOrCreate

And confirm the container mounts them correctly:

volumeMounts:
  - name: varlog
    mountPath: /var/log
    readOnly: true
  - name: varlibdockercontainers
    mountPath: /var/lib/docker/containers
    readOnly: true

Note: You mentioned /var/logs—this might be a typo; the standard path is /var/log.

3. Targets Are Being Added/Removed Repeatedly #

The logs showing Adding target followed immediately by Removing target usually happen when pods are restarting frequently, or Promtail’s Kubernetes service discovery is refreshing too often.

Fixes:

• Stabilize your pods: If your workloads are crashing or restarting constantly, resolve that first—frequent pod churn will keep Promtail in a loop of discovering and dropping targets.
• Adjust service discovery refresh interval: Add a refresh_interval to your kubernetes_sd_configs to reduce how often Promtail scans for pod changes:

  kubernetes_sd_configs:
    - role: pod
      refresh_interval: 30s
  

4. Verify Loki Connectivity (Just to Be Safe) #

While the main issue is on the Promtail side, double-check that Promtail can reach Loki:

• Exec into a Promtail pod and run: curl <your-loki-url>/ready
• If you get a 200 OK response, the connection is working. If not, check network policies, Loki service endpoints, or firewall rules blocking traffic.

Post-Fix Validation #

After making these changes:

1. Restart the Promtail DaemonSet: kubectl rollout restart daemonset promtail -n <your-namespace>
2. Access Promtail’s /targets endpoint—you should see active_targets: true for valid pods.
3. Check if /var/log/pods in the Promtail pod has log directories now.
4. Query Loki to confirm logs are flowing in.

内容的提问来源于stack exchange，提问作者SRK