AKS部署Jupyter-PySpark Notebook遇只读文件系统错误求助

问题描述 #

在Azure Kubernetes Services（AKS）部署jupyter-pyspark-notebook时，使用ConfigMap自定义jupyter_server_config.py以启用HTTPS，同时挂载PVC到/jupyter-storage。执行jupyter server list时出现以下错误：

OSError: [Errno 30] Read-only file system: '/home/jovyan/.jupyter/migrated'

Pod日志也显示无法写入/home/jovyan/.jupyter/lab，提示只读文件系统。

用到的配置文件 #

sts.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: jupyter-pyspark-notebook
  namespace: monitor-azure
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jupyter-pyspark-notebook
  template:
    metadata:
      labels:
        app: jupyter-pyspark-notebook
    spec:
      containers:
      - args:
        - bash
        - -c
        - |
          cd /jupyter-storage;
          for i in $(seq 1 $NUMBER_OF_NOTEBOOKS); do
            jupyter-lab --no-browser --port=$((8889 + i)) --ip=$HOSTNAME --allow-root &
          done
          while true; do
            sleep 10
          done      
        env:
        - name: NUMBER_OF_NOTEBOOKS
          value: "10"
        image: jupyter/pyspark-notebook
        imagePullPolicy: Always
        name: jupyter-container
        ports:
        - containerPort: 8890
          name: port1
          protocol: TCP
        - containerPort: 8891
          name: port2
          protocol: TCP
        - containerPort: 8892
          name: port3
          protocol: TCP
        - containerPort: 8893
          name: port4
          protocol: TCP
        - containerPort: 8894
          name: port5
          protocol: TCP
        - containerPort: 8895
          name: port6
          protocol: TCP
        - containerPort: 8896
          name: port7
          protocol: TCP
        - containerPort: 8897
          name: port8
          protocol: TCP
        - containerPort: 8898
          name: port9
          protocol: TCP
        - containerPort: 8899
          name: port10
          protocol: TCP
        resources:
          limits:
            cpu: "4"
            memory: 2Gi
          requests:
            cpu: "1"
            memory: 1Gi
        securityContext:
          runAsUser: 0
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /jupyter-storage
          name: jupyter-storage
        - name: jupyter-config-volume
          mountPath: /home/jovyan/.jupyter
        - name: jupyter-tls-secret-volume
          readOnly: true
          mountPath: /etc/jupyter             
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30
      tolerations:
        - key: monitor-azure
          operator: Equal
          value: dedicated
          effect: NoSchedule      
      volumes:
      - name: jupyter-storage
        persistentVolumeClaim:
          claimName: jupyter-storage
      - name: jupyter-config-volume
        configMap:
          name: jupyter-config
      - name: jupyter-tls-secret-volume
        secret:
          secretName: jupyter-tls 
  updateStrategy:
    rollingUpdate:
      partition: 0
type: RollingUpdate

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: jupyter-config
  namespace: monitor-azure
data:
  jupyter_server_config.py: |
    c.ServerApp.certfile = '/etc/jupyter/tls.crt'
    c.ServerApp.keyfile = '/etc/jupyter/tls.key'
    c.ServerApp.ip = '0.0.0.0'
    c.ServerApp.open_browser = False

Secret描述

C:\Users\test-user>kubectl -n monitor-azure describe secret jupyter-tls
Name:         jupyter-tls
Namespace:    monitor-azure
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.key:  1704 bytes
tls.crt:  1407 bytes

原因分析 #

当前配置将ConfigMap直接挂载到/home/jovyan/.jupyter整个目录，而Kubernetes中ConfigMap挂载的目录默认是只读的。Jupyter Lab启动时需要在该目录下创建migrated、lab等目录，用于存储动态配置、扩展缓存等内容，因此触发只读文件系统错误。

解决方案 #

不要将ConfigMap挂载到整个/home/jovyan/.jupyter目录，而是通过subPath仅挂载jupyter_server_config.py单个文件，保留原目录的可写权限。

修改后的StatefulSet配置（仅需调整volumeMounts部分） #

# sts.yaml中containers下的volumeMounts替换为以下内容
volumeMounts:
- mountPath: /jupyter-storage
  name: jupyter-storage
# 仅挂载单个配置文件，而非覆盖整个目录
- name: jupyter-config-volume
  mountPath: /home/jovyan/.jupyter/jupyter_server_config.py
  subPath: jupyter_server_config.py
- name: jupyter-tls-secret-volume
  readOnly: true
  mountPath: /etc/jupyter             

其他配置（ConfigMap、Secret、PVC）无需修改。

验证步骤 #

修改后重新部署StatefulSet：

kubectl apply -f sts.yaml -n monitor-azure

进入Pod执行jupyter server list或查看Pod日志，确认错误消失，Jupyter Lab实例可正常启动并写入必要文件。

补充说明 #

如果需要自定义更多Jupyter配置文件，可在ConfigMap中添加对应文件，然后通过多个subPath挂载到/home/jovyan/.jupyter目录下的对应位置，避免覆盖整个目录。

内容的提问来源于stack exchange，提问作者Tad