EKS使用EFS持久化存储访问目录报只读文件系统错误

现有部署配置 #

存储资源配置 #

----Storages files----
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
    name: aws-efs
provisioner: aws.io/aws-efs
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: efs-claim
namespace: dev
annotations:  
    volume.beta.kubernetes.io/storage-class: "aws-efs"
spec:
    accessModes:
        - ReadWriteMany
    resources:
        requests:
            storage: 20Gi

EFS Provisioner 相关配置 #

--------Deployment file--------------------
apiVersion: v1
kind: ServiceAccount
metadata:
  name: efs-provisioner
  namespace: dev
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: efs-provisioner
rules:
- apiGroups: [""]
  resources: ["persistentvolumes"]
  verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
  resources: ["storageclasses"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: efs-provisioner
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: efs-provisioner
subjects:
- kind: ServiceAccount
  name: efs-provisioner
  namespace: dev
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-efs-provisioner
  namespace: dev
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-efs-provisioner
  namespace: dev
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: leader-locking-efs-provisioner
subjects:
- kind: ServiceAccount
  name: efs-provisioner
  namespace: dev
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: efs-provisioner
  namespace: dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: efs-provisioner
  template:
    metadata:
      labels:
        app: efs-provisioner
    spec:
      serviceAccount: efs-provisioner
      containers:
      - name: efs-provisioner
        image: quay.io/external_storage/efs-provisioner:latest
        env:
        - name: FILE_SYSTEM_ID
          valueFrom:
            configMapKeyRef:
              name: efs-provisioner-config
              key: file.system.id
        - name: AWS_REGION
          valueFrom:
            configMapKeyRef:
              name: efs-provisioner-config
              key: aws.region
        - name: DNS_NAME
          valueFrom:
            configMapKeyRef:
              name: efs-provisioner-config
              key: dns.name
              optional: true
        - name: PROVISIONER_NAME
          valueFrom:
            configMapKeyRef:
              name: efs-provisioner-config
              key: provisioner.name
        volumeMounts:
        - name: pv-volume
          mountPath: /efs-mount
      volumes:
      - name: pv-volume
        nfs:
          server: <File-system-dns>
          path: /
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: efs-provisioner-config
  namespace: dev
data:
  file.system.id: <File-system-id>
  aws.region: us-east-2
  provisioner.name: aws.io/aws-efs
  dns.name: ""

Airflow Helm 发布配置 #

------release file----
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: airflow
  namespace: dev
  annotations:
    flux.weave.works/automated: "true"
spec:
  releaseName: airflow-dev
  chart:
    repository: https://airflow.apache.org
    name: airflow
    version: 1.6.0
  values:
    fernetKey: <fernet-key>
    defaultAirflowTag: "2.3.0"
    env:
      - name: "AIRFLOW__KUBERNETES__DAGS_IN_IMAGE"
        value: "False"
      - name: "AIRFLOW__KUBERNETES__NAMESPACE"
        value: "dev"
        value: "apache/airflow"
      - name: "AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG"
        value: "latest"
      - name: "AIRFLOW__KUBERNETES__RUN_AS_USER"
        value: "50000"
      - name: "AIRFLOW__CORE__LOAD_EXAMPLES"
        value: "False"
    executor: "KubernetesExecutor"
    dags:
      persistence:
        enabled: true
        size: 20Gi
        storageClassName: aws-efs
        existingClaim: efs-claim
        accessMode: ReadWriteMany
      gitSync:
        enabled: true
        repo: git@bitbucket.org: <git-repo>
        branch: master
        maxFailures: 0
        subPath: ""
        sshKeySecret: airflow-git-private-dags
        wait: 30

问题现象 #

• 进入scheduler Pod访问/opt/airflow/dags目录时，返回*只读文件系统（Read only file system）*错误
• 执行df -h可确认EFS文件系统已成功挂载至对应Pod内
• 执行kubectl get pv查看，对应PV的访问模式为RWX，状态显示已成功挂载至airflow trigger、airflow scheduler Pod

根因分析 #

1. EFS Provisioner组件版本过时：当前使用的quay.io/external_storage/efs-provisioner:latest为社区已停止维护的旧版本，对Kubernetes 1.21以上版本的RWX挂载逻辑存在兼容bug，创建PV时会隐式携带只读挂载参数，即使PVC层面显示RWX，实际kubelet执行挂载时仍会以只读模式加载卷。
2. 挂载配置冲突：Airflow Helm配置中同时开启了dags.persistence持久化存储和dags.gitSync代码同步两个特性，gitSync sidecar容器默认会以只读模式将拉取的Git仓库内容挂载到/opt/airflow/dags路径，同一路径被两个容器以不同权限挂载时，kubelet会统一降级为只读模式避免文件系统损坏。
3. 文件系统权限不匹配：配置中指定Airflow进程以UID 50000的用户运行，但EFS根目录默认属主为root用户，权限为755，运行用户无目录写入权限，部分Linux内核版本会直接将无写入权限的挂载点标记为只读返回给应用层。
4. 配置语法错误：Helm values的env配置块中，AIRFLOW__KUBERNETES__NAMESPACE变量下重复定义了value: "apache/airflow"字段，会导致Pod环境变量解析异常，部分挂载参数无法正确传递给kubelet。
5. 网络策略拦截：若EFS挂载安全组未对EKS工作节点完全开放2049（NFS）端口的读写权限，NFS客户端会自动降级为只读挂载，避免写入操作失败导致数据异常。

解决方案 #

1. 替换过时的EFS Provisioner
   • 清理现有旧版efs-provisioner对应的Deployment、ServiceAccount、ClusterRole、ClusterRoleBinding等所有资源
   • 部署AWS官方维护的EFS CSI驱动，为驱动配置对应EFS操作权限的IAM角色
   • 重新创建StorageClass，指定正确的权限配置：

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: aws-efs
provisioner: efs.csi.aws.com
parameters:
  fileSystemId: <替换为实际EFS文件系统ID>
  directoryPerms: "775"
  gidRangeStart: "1000"
  gidRangeEnd: "65535"
  basePath: "/airflow_dags"

2. 修复Airflow配置冲突
   • 二选一配置DAG加载方式：若使用EFS持久化存储存放DAG，关闭gitSync特性；若需要通过Git同步DAG，将持久化卷挂载到非DAG目录路径，避免同一路径多容器挂载权限冲突
   • 删除env配置块中重复的无效value: "apache/airflow"行，修正环境变量语法
   • 在dags.persistence配置块下显式添加readOnly: false参数，明确指定卷挂载为读写模式
3. 修复文件系统权限
   • 在EFS控制台创建专用访问点，配置POSIX用户为UID 50000、GID 50000，目录权限为775，创建PV时关联该访问点
   • 临时将EFS挂载到同VPC下的EC2实例，将DAG存储目录的属主修改为50000:50000，确保Airflow运行用户有读写权限
4. 验证网络连通性
   • 检查EFS关联的安全组入站规则，确保EKS工作节点所属安全组可访问2049端口，无网络ACL或安全组拦截NFS读写请求
   • 重新创建PVC、PV，重启所有Airflow相关Pod，进入scheduler容器执行touch /opt/airflow/dags/test命令，验证写入功能正常

内容的提问来源于stack exchange，提问作者bhavesh singh