Kubernetes集群中MariaDB Deployment多副本反复崩溃重启问题排查求助

大家好，我在Kubernetes集群部署MariaDB时碰到了棘手的问题：我创建了一个带3个副本的Deployment，但启动30秒后，有2个副本会崩溃、重启，然后再次陷入崩溃循环。看起来问题和绑定的PVC脱不了干系。

从现象来看，Pod似乎需要独占PV里的某些核心文件，目前只有一个Pod能正常运行。有意思的是，类似的多副本配置在InfluxDB和Grafana上都能正常工作，唯独MariaDB出了问题。

我的部署配置文件 #

apiVersion: apps/v1
kind: Deployment
metadata:
name: mariadb
labels:
app: mariadb
spec:
replicas: 3
selector:
matchLabels:
app: mariadb
strategy:
type: Recreate
template:
metadata:
labels:
app: mariadb
spec:
containers:
- name: mariadb
image: mariadb:latest
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
key: MYSQL_ROOT_PASSWORD
name: platform-secret
[more variables]
volumeMounts:
- mountPath: /var/lib/mysql
name: mariadb-pv
restartPolicy: Always
volumes:
- name: mariadb-pv
persistentVolumeClaim:
claimName: mariadb-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mariadb-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
labels:
app: mariadb
name: mariadb
spec:
selector:
app: influxdb
type: ClusterIP
ports:
- protocol: TCP
port: 3306
targetPort: 3306
status:
loadBalancer: {}

我尝试过的解决方法 #

我试过拆分PVC，给/var/lib/mysql下的子文件夹分别挂载独立的Volume，这种方式是可行的：

volumeMounts:
- name: mariadb-aria-storage
  mountPath: /var/lib/mysql/aria
- name: mariadb-innodb-storage
  mountPath: /var/lib/mysql/innodb

但直接把整个/var/lib/mysql挂载到同一个PVC时，问题就会出现。

相关错误指标 #

1. Pod状态：

kubectl get pods
NAME                       READY   STATUS    RESTARTS      AGE
mariadb-78d7945894-gsjnq   1/1     Running   2 (28s ago)   103s
mariadb-78d7945894-qzfqd   1/1     Running   0             103s
mariadb-78d7945894-zmg7k   1/1     Running   2 (27s ago)   103s

2. 事件日志（关键片段）：

kubectl get event
[...]
2m10s       Warning   FailedAttachVolume       pod/mariadb-78d7945894-zmg7k        AttachVolume.Attach failed for volume "pvc-c5b09340-5060-4b77-921a-9f889bc81e84" : rpc error: code = Internal desc = [ControllerPublishVolume] Attach Volume failed with error failed to attach e162b4b1-be0f-4b63-a8eb-6bf679c9719e volume to cfb3d471-5184-4a79-8474-dc62c4893c7d compute: Bad request with: [POST https://ops.elastx.cloud:8774/v2.1/servers/cfb3d471-5184-4a79-8474-dc62c4893c7d/os-volume_attachments], error message: {"badRequest": {"code": 400, "message": "Invalid input received: Invalid volume: Volume e162b4b1-be0f-4b63-a8eb-6bf679c9719e status must be available or downloading to reserve, but the current status is creating. (HTTP 400) (Request-ID: req-6f2d38f9-c0d4-43ab-81bf-da75d347e723)"}}
2m17s       Warning   FailedScheduling         pod/mariadb-78d7945894-zmg7k        running PreBind plugin "VolumeBinding": Operation cannot be fulfilled on persistentvolumeclaims "mariadb-pvc": the object has been modified; please apply your changes to the latest version and try again
1s          Warning   BackOff                  pod/mariadb-78d7945894-zmg7k        Back-off restarting failed container mariadb in pod mariadb-78d7945894-zmg7k_default(d0952758-94e9-4d9e-aeda-c8f851a891b6)
[...]

3. 崩溃Pod的日志：

kubectl logs mariadb-78d7945894-zmg7k
2023-11-30 13:31:58+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server latest started.
2023-11-30 13:31:58+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2023-11-30 13:31:58+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server latest started.
2023-11-30 13:31:58+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
2023-11-30 13:31:58 0 [Note] Starting MariaDB 11.2.2-MariaDB-1:11.2.2+maria~ubu2204 source revision 929532a9426d085111c24c63de9c23cc54382259 as process 1
2023-11-30 13:31:58 0 [ERROR] mariadbd: Can't lock aria control file '/var/lib/mysql/aria_log_control' for exclusive use, error: 11. Will retry for 30 seconds
2023-11-30 13:32:28 0 [ERROR] mariadbd: Got error 'Could not get an exclusive lock; file is probably in use by another process' when trying to use aria control file '/var/lib/mysql/aria_log_control'
2023-11-30 13:32:28 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
2023-11-30 13:32:28 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-11-30 13:32:28 0 [Note] InnoDB: Number of transaction pools: 1
2023-11-30 13:32:28 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-11-30 13:32:28 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2023-11-30 13:32:28 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2023-11-30 13:32:28 0 [Note] InnoDB: Completed initialization of buffer pool
2023-11-30 13:32:28 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
2023-11-30 13:32:28 0 [Note] InnoDB: Check that you do not already have another mariadbd process using the same InnoDB data or log files.
2023-11-30 13:32:28 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2023-11-30 13:32:28 0 [Note] InnoDB: Starting shutdown...
2023-11-30 13:32:28 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2023-11-30 13:32:28 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-11-30 13:32:28 0 [Note] Plugin 'wsrep-provider' is disabled.
2023-11-30 13:32:28 0 [ERROR] Could not open mysql.plugin table: "Unknown storage engine 'Aria'". Some plugins may be not loaded
2023-11-30 13:32:28 0 [ERROR] Failed to initialize plugins.
2023-11-30 13:32:28 0 [ERROR] Aborting

有没有大佬能帮我分析下问题根源？如果需要更多信息，随时告诉我！

备注：内容来源于stack exchange，提问作者janton_neab