AppArmor Complain Mode绑定Kubernetes Pod时未按预期生效的问题咨询

我最近在测试AppArmor的Complain模式和Kubernetes结合的场景时遇到了一个奇怪的问题，想请教大家这是AppArmor的bug还是我哪里操作有误？

我的操作步骤 #

1. 创建AppArmor Profile #

我创建了一个禁止所有文件写入的profile，路径是/etc/apparmor.d/test-deny-write，内容如下：

#include <tunables/global>

profile test-deny-write flags=(attach_disconnected) {

#include <abstractions/base>

file,

# Deny all file writes.

deny /** w,

}

2. 以Complain模式加载Profile #

执行以下命令加载profile并查看状态：

sudo apparmor_parser --Complain test-deny-write
sudo aa-status

aa-status的输出明确显示test-deny-write处于complain模式：

apparmor module is loaded.
38 profiles are loaded.
31 profiles are in enforce mode.
/snap/snapd/20290/usr/lib/snapd/snap-confine
/snap/snapd/20290/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
/usr/bin/man
/usr/lib/NetworkManager/nm-dhcp-client.action
/usr/lib/NetworkManager/nm-dhcp-helper
/usr/lib/connman/scripts/dhclient-script
/usr/lib/snapd/snap-confine
/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
/usr/sbin/chronyd
/usr/sbin/tcpdump
/{,usr/}sbin/dhclient
cri-containerd.apparmor.d
lsb_release
man_filter
man_groff
nvidia_modprobe
nvidia_modprobe//kmod
snap-update-ns.google-cloud-cli
snap-update-ns.lxd
snap.lxd.activate
snap.lxd.benchmark
snap.lxd.buginfo
snap.lxd.check-kernel
snap.lxd.daemon
snap.lxd.hook.configure
snap.lxd.hook.install
snap.lxd.hook.remove
snap.lxd.lxc
snap.lxd.lxc-to-lxd
snap.lxd.lxd
snap.lxd.migrate
7 profiles are in complain mode.
snap.google-cloud-cli.anthoscli
snap.google-cloud-cli.bq
snap.google-cloud-cli.docker-credential-gcloud
snap.google-cloud-cli.gcloud
snap.google-cloud-cli.gsutil
snap.google-cloud-cli.kubectl
test-deny-write
13 processes have profiles defined.
13 processes are in enforce mode.
/usr/sbin/chronyd (1376)
/usr/sbin/chronyd (1377)
/metrics-server (12860) cri-containerd.apparmor.d
/usr/bin/dumb-init (13041) cri-containerd.apparmor.d
/nginx-ingress-controller (13053) cri-containerd.apparmor.d
/manager (13086) cri-containerd.apparmor.d
/usr/local/nginx/sbin/nginx (13114) cri-containerd.apparmor.d
/manager (13136) cri-containerd.apparmor.d
/usr/local/nginx/sbin/nginx (13155) cri-containerd.apparmor.d
/usr/local/nginx/sbin/nginx (13156) cri-containerd.apparmor.d
/usr/local/nginx/sbin/nginx (13157) cri-containerd.apparmor.d
/manager (13250) cri-containerd.apparmor.d
/manager (13565) cri-containerd.apparmor.d
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

3. 部署绑定该Profile的Kubernetes Pod #

我定义了一个绑定该AppArmor profile的Nginx Pod，配置文件n.yaml内容如下：

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: nginx
  annotations:
    container.apparmor.security.beta.kubernetes.io/nginx: localhost/test-deny-write
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx
    ports:
    - containerPort: 80
  dnsPolicy: ClusterFirst
  restartPolicy: Always

执行部署命令：

kubectl apply -f n.yaml

但Pod启动失败，进入CrashLoopBackOff状态：

kubectl get pods
NAME                              READY   STATUS             RESTARTS     AGE
...
nginx                             0/1     CrashLoopBackOff   1 (3s ago)   7s

查看Pod日志发现是权限被拒绝，这说明AppArmor实际上是在enforce模式下生效的，完全不符合我预期的Complain模式行为：

/docker-entrypoint.sh: 13: cannot create /dev/null: Permission denied
/docker-entrypoint.sh: No files found in /docker-entrypoint.d/, skipping configuration
2024/01/06 11:27:50 [emerg] 1#1: mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)
nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)

4. 修改Profile为Audit规则后的情况 #

当我把Profile里的deny /** w,改成audit /** w,，然后重新以Complain模式加载：

#include <tunables/global>

profile test-deny-write flags=(attach_disconnected) {

#include <abstractions/base>

file,

# Audit all file writes.

audit /** w,

}

重新加载后，Pod可以正常启动了，并且写入操作会被正常审计。比如我在Pod内执行写入命令：

kubectl exec -it nginx -- sh
# echo a > a.log

查看dmesg能看到对应的审计日志：

[ 1943.792266] audit: type=1400 audit(1704541508.355:101): apparmor="AUDIT" operation="open" profile="test-deny-write" name="/dev/tty" pid=16159 comm="sh" requested_mask="w" fsuid=0 ouid=0
[ 1950.228525] audit: type=1400 audit(1704541514.792:102): apparmor="AUDIT" operation="mknod" profile="test-deny-write" name="/a.log" pid=16159 comm="sh" requested_mask="c" fsuid=0 ouid=0
[ 1950.234413] audit: type=1400 audit(1704541514.800:103): apparmor="AUDIT" operation="open" profile="test-deny-write" name="/a.log" pid=16159 comm="sh" requested_mask="wc" fsuid=0 ouid=0

但这样的配置完全失去了Complain模式的意义——原本应该只记录违规操作而不阻止，现在变成了单纯的审计。

系统环境信息 #

• 内核版本：Linux node-qf1h 5.15.0-1047-gcp #55~20.04.1-Ubuntu SMP Wed Nov 15 11:38:25 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
• Kubernetes版本：

  Client Version: v1.28.0
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.28.0
  

我查过AppArmor的相关文档，里面关于Complain模式和deny规则的部分语焉不详，只提到部分版本可能存在问题，但没有具体说明。

想请教各位，这是AppArmor的已知bug吗？还是我在配置或者操作过程中遗漏了什么关键步骤？

备注：内容来源于stack exchange，提问作者user1069483