容器运行MySQL时数据库频繁丢失的原因排查

环境与问题描述 #

• 运行环境：DigitalOcean 2GB内存Droplet，通过Laravel Sail部署Laravel Web应用
• 问题现象：应用运行超过12小时后（时间不固定），访问时出现数据库不存在错误：

Illuminate\Database\QueryException
SQLSTATE[HY000] [1049] Unknown database 'laravel'
select * from sessions where id = NScfhR1av3SqPJWO658qfYzrevdBmtpB7Sd7801z limit 1

手动重启容器后数据库恢复正常，监控显示剩余内存900MB以上，排除内存不足问题。

当前MySQL容器配置（docker-compose.yml） #

mysql:
        image: 'mysql/mysql-server:8.0'
        restart: unless-stopped
        ports:
            - '${FORWARD_DB_PORT:-3306}:3306'
        environment:
            MYSQL_ROOT_PASSWORD: '${DB_PASSWORD}'
            MYSQL_ROOT_HOST: '%'
            MYSQL_DATABASE: '${DB_DATABASE}'
            MYSQL_USER: '${DB_USERNAME}'
            MYSQL_PASSWORD: '${DB_PASSWORD}'
            MYSQL_ALLOW_EMPTY_PASSWORD: 1
            MYSQL_LOG_CONSOLE: "true"
        command: [
            "--general-log=1",
            "--general-log-file=/var/lib/mysql/general.log"
        ]
        volumes:
            - 'sail-mysql:/var/lib/mysql'
        networks:
            - sail
        healthcheck:
            test: ["CMD", "mysqladmin", "ping", "-p${DB_PASSWORD}"]
            interval: 30s
            timeout: 10s
            retries: 5

相关日志 #

MySQL容器重启片段日志 #

mysql_1         | 2025-02-26T11:10:44.197336Z 963 [System] [MY-013172] [Server] Received SHUTDOWN from user root. Shutting down mysqld (Version: 8.0.32).
mysql_1         | 2025-02-26T11:10:45.622896Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.32)  MySQL Community Server - GPL.
mysql_1         | [Entrypoint] MySQL Docker Image 8.0.32-1.2.11-server
mysql_1         | [Entrypoint] Starting MySQL 8.0.32-1.2.11-server
mysql_1         | 2025-02-26T11:10:46.474184Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
mysql_1         | 2025-02-26T11:10:46.476331Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.32) starting as process 1
mysql_1         | 2025-02-26T11:10:46.487273Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
mysql_1         | 2025-02-26T11:10:46.701850Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
mysql_1         | 2025-02-26T11:10:46.893607Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
mysql_1         | 2025-02-26T11:10:46.893885Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
mysql_1         | 2025-02-26T11:10:46.926131Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
mysql_1         | 2025-02-26T11:10:46.926655Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.32'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server

Docker服务日志（journalctl片段） #

Feb 26 04:09:00 manu-connect dockerd[1137]: time="2025-02-26T04:09:00.414881863Z" level=error msg="stream copy error: reading from a closed fifo"
Feb 26 04:09:00 manu-connect dockerd[1137]: time="2025-02-26T04:09:00.414973931Z" level=error msg="stream copy error: reading from a closed fifo"
Feb 26 04:09:00 manu-connect dockerd[1137]: time="2025-02-26T04:09:00.448979113Z" level=info msg="ignoring event" container=fbe008fee27623c5918b0b91d1aa7bac86bbc0343a4b11fc5d2250e5af05f307 module=libcontainer>
Feb 26 04:09:00 manu-connect dockerd[1137]: time="2025-02-26T04:09:00.458393016Z" level=warning msg="Health check for container fbe008fee27623c5918b0b91d1aa7bac86bbc0343a4b11fc5d2250e5af05f307 error: context >
Feb 26 04:09:00 manu-connect dockerd[1137]: time="2025-02-26T04:09:00.850173433Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers"
Feb 26 11:10:45 manu-connect dockerd[1137]: time="2025-02-26T11:10:45.819352426Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers"
Feb 26 12:12:56 manu-connect dockerd[1137]: 2025/02/26 12:12:56 http: superfluous response.WriteHeader call from go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*respWriterWrapper).WriteHeader >
Feb 26 12:15:54 manu-connect dockerd[1137]: 2025/02/26 12:15:54 http: superfluous response.WriteHeader call from go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*respWriterWrapper).WriteHeader >

配置问题排查 #

• 环境变量冲突：配置中同时设置MYSQL_ROOT_PASSWORD和MYSQL_ALLOW_EMPTY_PASSWORD: 1，二者存在冲突。MySQL官方镜像中，当MYSQL_ALLOW_EMPTY_PASSWORD设为1时会忽略MYSQL_ROOT_PASSWORD，可能导致容器重启时身份验证异常，触发初始化逻辑重置数据库。建议移除MYSQL_ALLOW_EMPTY_PASSWORD: 1，仅保留MYSQL_ROOT_PASSWORD。
• 健康检查命令缺陷：当前健康检查命令mysqladmin ping -p${DB_PASSWORD}若密码包含特殊字符，会被shell解析错误，导致健康检查失败。可修改为mysqladmin ping -u root -p"${DB_PASSWORD}"，明确指定root用户并给密码加引号避免解析问题。
• 卷挂载有效性验证：执行docker volume inspect sail-mysql查看卷的主机挂载路径，检查对应目录下是否存在MySQL数据文件（如ibdata1、laravel文件夹）。若卷被意外删除或未正确挂载，容器重启时会重新初始化数据库。

其他排查方向 #

• 分析MySQL通用日志：已开启的通用日志路径为/var/lib/mysql/general.log，进入容器（docker exec -it <mysql容器ID> bash）查看日志，确认数据库删除操作的来源（内部命令或外部请求）。
• 排查容器异常重启原因：通过docker events --since 24h查看Docker事件日志，检查是否有容器被强制停止、重启的记录；同时查看Droplet系统日志（/var/log/syslog或/var/log/messages），排查是否存在OOM killer、磁盘空间不足或系统重启情况。
• 检查磁盘空间：执行df -h确认Droplet磁盘剩余空间，若磁盘满会导致MySQL无法写入数据，甚至重启时出现数据损坏或丢失。
• 排查应用代码逻辑：检查Laravel应用中是否存在意外删除数据库的代码，比如迁移脚本、定时任务（Laravel Scheduler）或第三方包操作。
• 验证MySQL版本兼容性：当前使用的MySQL 8.0.32是否存在容器重启后数据丢失的已知bug？可尝试升级到最新8.0.x版本或降级到稳定版本测试。

内容的提问来源于stack exchange，提问作者NicLovin