K8s中MongoDB 5.0.12 Pod启动失败：Operation not permitted排查求助

问题描述 #

在Kubernetes中运行MongoDB 5.0.12实例作为Pod，Pod突然启动失败，日志如下：

{"t":{"$date":"2022-09-13T18:39:51.104+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"AuthorizationManager-1","msg":"WiredTiger error","attr":{"error":1,"message":"[1663094391:104664][1:0x7fc5224cc700], file:index-9--3195476868760592993.wt, WT_SESSION.open_cursor: __posix_open_file, 808: /data/db/index-9--3195476868760592993.wt: handle-open: open: Operation not permitted"}}
{"t":{"$date":"2022-09-13T18:39:51.104+00:00"},"s":"F",  "c":"STORAGE",  "id":50882,   "ctx":"AuthorizationManager-1","msg":"Failed to open WiredTiger cursor. This may be due to data corruption","attr":{"uri":"table:index-9--3195476868760592993","config":"overwrite=false","error":{"code":8,"codeName":"UnknownError","errmsg":"1: Operation not permitted"},"message":"Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair"}}
{"t":{"$date":"2022-09-13T18:39:51.104+00:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"AuthorizationManager-1","msg":"Fatal assertion","attr":{"msgid":50882,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp","line":109}}
{"t":{"$date":"2022-09-13T18:39:51.104+00:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"AuthorizationManager-1","msg":"\n\n***aborting after fassert() failure\n\n"}

已执行mongod --repair但问题仍存在，需排查错误原因及解决方法。

部署配置 #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongodb
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      hostname: mongodb
      # securityContext:
      #   runAsUser: 999
      #   runAsGroup: 3000
      #   fsGroup: 2000
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: data
      containers:
        - name: mongodb
          image: mongo:5.0.12
          args: ["--auth", "--dbpath", "/data/db"]
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 27017
          volumeMounts:
            - mountPath: /data/db
              name: data
          # securityContext:
          #   allowPrivilegeEscalation: false

PVC配置 #

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

错误原因分析 #

核心问题是MongoDB进程无法打开WiredTiger索引文件，提示Operation not permitted，结合配置分析主要原因：

• 官方mongo:5.0.12镜像默认使用mongodb用户（UID=999，GID=999）运行mongod进程
• 部署中注释了Pod的securityContext配置，导致容器以root用户启动，但PV挂载的/data/db目录及文件可能是之前用非root用户创建的，或PV权限与当前运行用户不匹配
• 执行mongod --repair时用户权限错误，修复操作无法正确修改文件权限或修复数据，因此问题未解决

解决步骤 #

1. 启用匹配镜像用户的安全上下文 #

取消Deployment中securityContext的注释，修改为匹配镜像默认用户的配置：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongodb
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      hostname: mongodb
      securityContext:
        runAsUser: 999
        runAsGroup: 999
        fsGroup: 999
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: data
      containers:
        - name: mongodb
          image: mongo:5.0.12
          args: ["--auth", "--dbpath", "/data/db"]
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 27017
          volumeMounts:
            - mountPath: /data/db
              name: data

说明：runAsUser和runAsGroup设置为999，匹配镜像默认的mongodb用户；fsGroup会确保PV挂载目录的文件组权限为999，让mongod进程拥有读写权限

2. 手动修复PV目录权限（若步骤1无效） #

如果启用安全上下文后仍报错，需手动修正PV中/data/db的权限：

• 创建临时特权Pod挂载同一PVC：

apiVersion: v1
kind: Pod
metadata:
  name: fix-mongo-perms
spec:
  containers:
  - name: busybox
    image: busybox:latest
    command: ["sh", "-c", "chown -R 999:999 /data/db && sleep 3600"]
    volumeMounts:
    - name: data
      mountPath: /data/db
    securityContext:
      privileged: true
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: data

• 启动Pod后等待权限修改完成，删除临时Pod，再重启MongoDB Deployment

3. 权限正确后重新执行修复（可选） #

若数据确实存在损坏，在权限正常的前提下执行修复：

# 进入MongoDB Pod
kubectl exec -it <mongodb-pod-name> -- bash
# 执行修复操作
mongod --dbpath /data/db --repair

修复完成后重启Pod即可

内容的提问来源于stack exchange，提问作者user3142695