Kubernetes中MySQL 5.6 Pod启动失败，请求排查故障原因

问题背景 #

使用以下Deployment部署MySQL 5.6容器，Pod多次重启仍无法正常启动：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql-1
  namespace: test
  resourceVersion: "123213"
  uid: xxxxxx
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: mysql
      tier: mysql-1
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: mysql
        tier: mysql-1
    spec:
      containers:
      - env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: rootPassword
              name: mysql-1-password
        image: mysql:5.6
        imagePullPolicy: IfNotPresent
        name: mysql-1
        ports:
        - containerPort: 3306
          name: mysql
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/mysql
          name: mysql-persistent-storage
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-56-1-pvc

Pod启动失败日志：

2023-09-26 10:36:31 0 [Note] mysqld (mysqld 5.6.51) starting as process 1 ...
2023-09-26 10:36:31 1 [Note] Plugin 'FEDERATED' is disabled.
mysqld: Table 'mysql.plugin' doesn't exist
2023-09-26 10:36:31 1 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
2023-09-26 10:36:31 1 [Note] InnoDB: Using atomics to ref count buffer pool pages
2023-09-26 10:36:31 1 [Note] InnoDB: The InnoDB memory heap is disabled
2023-09-26 10:36:31 1 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-09-26 10:36:31 1 [Note] InnoDB: Memory barrier is not used
2023-09-26 10:36:31 1 [Note] InnoDB: Compressed tables use zlib 1.2.11
InnoDB: Warning: io_setup() attempt 1-5 failed.
2023-09-26 10:36:34 7f2030a71040 InnoDB: Error: io_setup() failed with EAGAIN after 5 attempts.
InnoDB: You can disable Linux Native AIO by setting innodb_use_native_aio = 0 in my.cnf
2023-09-26 10:36:34 1 [ERROR] InnoDB: Fatal : Cannot initialize AIO sub-system
2023-09-26 10:36:34 1 [ERROR] Plugin 'InnoDB' init function returned error.
2023-09-26 10:36:34 1 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2023-09-26 10:36:34 1 [ERROR] Unknown/unsupported storage engine: InnoDB
2023-09-26 10:36:34 1 [ERROR] Aborting

2023-09-26 10:36:34 1 [Note] Binlog end

错误分析 #

1. 核心致命错误：InnoDB AIO子系统初始化失败
   日志中明确提示InnoDB: Error: io_setup() failed with EAGAIN after 5 attempts，这是导致MySQL启动失败的直接原因。Linux Native AIO在部分Kubernetes存储环境（如某些分布式存储、本地存储配置）中可能存在兼容性问题，导致无法初始化AIO子系统，进而引发InnoDB引擎注册失败。

2. mysql.plugin表不存在是衍生错误
   Table 'mysql.plugin' doesn't exist是因为InnoDB引擎启动失败，MySQL无法完成系统库的初始化或加载，并非需要执行mysql_upgrade，这个提示属于误导性信息，根源还是InnoDB的启动问题。

解决方案 #

方法一：禁用Linux Native AIO #

在MySQL配置中添加innodb_use_native_aio = 0来绕过AIO兼容性问题：

1. 创建MySQL配置文件my.cnf：

[mysqld]
innodb_use_native_aio = 0

2. 通过ConfigMap将配置文件挂载到Pod中：

apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-config
  namespace: test
data:
  my.cnf: |
    [mysqld]
    innodb_use_native_aio = 0

3. 修改Deployment，添加ConfigMap挂载：
   在容器的volumeMounts中添加：

- mountPath: /etc/mysql/conf.d
  name: mysql-config

在Pod的volumes中添加：

- name: mysql-config
  configMap:
    name: mysql-config

方法二：检查存储卷权限与状态 #

• 确认PVCmysql-56-1-pvc绑定的PV状态正常，存储介质可正常读写。
• 检查Pod容器对/var/lib/mysql目录的权限，MySQL运行用户（默认是mysql:mysql）需要该目录的读写权限。可以通过在Deployment中添加securityContext设置权限：

securityContext:
  runAsUser: 999
  runAsGroup: 999
  fsGroup: 999

方法三：重置存储卷（仅适用于无数据场景） #

如果存储卷中没有需要保留的数据，可以删除PVC并重新创建，让MySQL重新初始化系统库：

kubectl delete pvc mysql-56-1-pvc -n test
# 等待PVC删除后，重新应用Deployment，PVC会自动重建
kubectl apply -f deployment.yaml -n test

内容的提问来源于stack exchange，提问作者Zexxxx