Falco Kubernetes审计规则无告警排查求助：已按文档配置但未触发告警

Let’s walk through a step-by-step troubleshooting process to figure out why your Falco audit rules aren’t triggering alerts:

1. Verify Audit Config Files Are Synced to Minikube Node #

First, confirm the files you placed in ~/.minikube/files/etc/ssl/certs exist inside the minikube node with the correct content:

# Enter the minikube node shell
minikube ssh

# Check if both config files are present
ls /etc/ssl/certs/audit-policy.yaml /etc/ssl/certs/audit-webhook-config.yaml

# Validate file content matches what you created
cat /etc/ssl/certs/audit-policy.yaml

If files are missing or content is incorrect, double-check the path in ~/.minikube/files (this directory syncs directly to the root / of the minikube node, so typos in the path are a common culprit).

2. Check Apiserver Audit Config Loading & Logs #

Next, ensure the kube-apiserver successfully loaded your audit policy and webhook config. Search the apiserver logs for audit-related messages:

kubectl logs kube-apiserver-minikube -n kube-system | grep -i audit

Look for confirmation lines like:

• Successfully loaded audit policy (validates your policy is syntactically correct)
• Loaded audit webhook config (confirms the webhook config was recognized)
  If you see errors like audit policy file not found or invalid YAML, fix the file path or syntax issues immediately.

3. Test Webhook Reachability from the Apiserver #

Your audit webhook points to http://127.0.0.1:32765/k8s-audit, but we need to confirm the apiserver can reach this endpoint inside the minikube node:

# Run this from inside the minikube node (after running minikube ssh)
curl -v http://127.0.0.1:32765/k8s-audit

A 405 Method Not Allowed response means the endpoint is reachable (Falco is listening). If you get a connection refused error:

• If Falco runs as a pod, ensure it uses a hostPort: 32765 in its deployment, or that a NodePort service maps to Falco’s audit port (default is 8765, so adjust your service or webhook URL if needed).
• Verify Falco’s main config (/etc/falco/falco.yaml) has k8s_audit.enabled: true and http_server.enabled: true (the audit endpoint uses the HTTP server port).

4. Validate Falco’s Audit Configuration & Rule Loading #

Make sure Falco is set up to receive and process Kubernetes audit events:

1. Check Falco’s core config (exec into the Falco pod if running in the cluster):

   # Example valid config snippet
   k8s_audit:
     enabled: true
   http_server:
     enabled: true
     port: 32765  # Must match the port in your audit webhook config
   

2. Verify the k8s audit rules load without errors:

   # For a local Falco instance
   falco -r /etc/falco/rules/k8s-audit-rules.yaml --dry-run
   
   # For a Falco pod
   kubectl exec -it <falco-pod-name> -n falco -- falco -r /etc/falco/rules/k8s-audit-rules.yaml --dry-run
   

   This command will flag any syntax errors in the rules that could prevent triggering.

5. Generate a Test Event to Validate End-to-End Flow #

Let’s create an action that should trigger a Falco audit rule to test the full pipeline:

1. Create a test cluster role (this should trigger the K8s Audit ClusterRole Created rule):

   kubectl create clusterrole test-falco-audit --verb=get --resource=pods
   

2. Check Falco’s logs immediately after:

   kubectl logs <falco-pod-name> -n falco | grep "ClusterRole Created"
   

3. Check if the apiserver sent the event using metrics:

   kubectl get --raw /metrics | grep audit_webhook
   

   Look for audit_webhook_requests_total{code="200"}—this count should increase after your test command, confirming the apiserver successfully sent the event to Falco.

6. Confirm Audit Policy Captures Required Events #

Double-check your audit-policy.yaml to ensure it captures the events needed for Falco’s rules. For example, the ClusterRole Created rule requires a RequestResponse or Request level event for clusterroles resources—your policy already includes this:

- level: RequestResponse
  resources:
  - group: "rbac.authorization.k8s.io"
    resources: ["clusterroles", "clusterrolebindings"]

If testing other rules, ensure the audit policy covers those resources and audit levels.

内容的提问来源于stack exchange，提问作者Sathya