MLFlow容器无法连接MySQL数据库的排查与解决求助

问题描述 #

制作了MLFlow Docker镜像后，启动容器时无法连接另一个Docker镜像中的MySQL数据库。启动MLFlow容器的脚本片段如下：

...
DB_BACKEND="mysql+pymysql://${MLFLOW_USER}:${MLFLOW_PASS}@$IP:${MYSQL_PORT1}/mlflow"
MLFLOW_CMD="mlflow server --backend-store-uri ${DB_BACKEND} --artifacts-destination /mlartifacts --serve-artifacts -h 0.0.0.0 -p 5000"

docker run --rm \
  --name mlflow-server \
  -p 5000:5000 \
  --volume ${DIR_SERVER}/artifacts:/mlartifacts \
  -d mlflow-ubuntu-22.04:22110701 \
  ${MLFLOW_CMD}

容器日志显示连接拒绝错误：

$ docker logs mlflow-server
2022/11/08 15:33:34 WARNING mlflow.store.db.utils: SQLAlchemy engine could not be created. The following exception is caught.
(pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '172.17.0.2' ([Errno 111] Connection refused)")
(Background on this error at: https://sqlalche.me/e/14/e3q8)
Operation will be retried in 0.1 seconds

但进入MLFlow容器后，使用mysql命令可以正常连接该MySQL数据库，且拥有建表等操作权限：

$ docker exec -it mlflow-server bash
root@b68c447b5ce9:/# mysql -u mlflow -p -h 172.17.0.2 mlflow
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2413
Server version: 8.0.31 MySQL Community Server - GPL
Copyright (c) 2000, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE TABLE test(id int);
Query OK, 0 rows affected (0.91 sec)

mysql> show tables;
+------------------+
| Tables_in_mlflow |
+------------------+
| test             |
+------------------+
1 row in set (0.01 sec)

mysql> drop table test;
Query OK, 0 rows affected (0.41 sec)

mysql>

修复方法 #

1. 确保MySQL服务在MLFlow启动前完全就绪 #

Docker容器启动顺序不保证依赖服务的就绪状态，MySQL启动需要加载数据、初始化服务，可能比MLFlow慢。可以：

• 使用docker-compose管理容器，通过depends_on结合健康检查，确保MLFlow仅在MySQL服务就绪后启动。示例docker-compose.yml片段：

  services:
    mysql:
      image: mysql:8.0.31
      healthcheck:
        test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
        interval: 5s
        timeout: 5s
        retries: 5
    mlflow:
      image: mlflow-ubuntu-22.04:22110701
      depends_on:
        mysql:
          condition: service_healthy
  

• 手动在启动脚本中添加等待逻辑，比如使用wait-for-it.sh脚本等待MySQL端口开放后再启动MLFlow：

  ./wait-for-it.sh $IP:${MYSQL_PORT1} --timeout=30 -- ${MLFLOW_CMD}
  

2. 使用Docker容器名称替代IP地址（推荐） #

默认Docker桥接网络的IP可能随容器重启变化，且容器间通过名称解析更可靠：

• 确保MLFlow和MySQL容器在同一个自定义Docker网络中（若未使用默认网络），先创建网络：docker network create mlflow-network，启动容器时添加--network mlflow-network参数。
• 修改连接字符串中的IP为MySQL容器名称，比如MySQL容器名为mysql-server，则：

  DB_BACKEND="mysql+pymysql://${MLFLOW_USER}:${MLFLOW_PASS}@mysql-server:${MYSQL_PORT1}/mlflow"
  

3. 验证连接字符串的正确性 #

确认启动脚本中的环境变量是否正确传递：

• 在启动MLFlow容器前，打印DB_BACKEND变量值，检查用户名、密码、IP、端口是否正确：

  echo ${DB_BACKEND}
  

• 进入MLFlow容器后，直接打印该变量，确认容器内获取到的连接字符串无误：

  docker exec mlflow-server echo $DB_BACKEND
  

调试方法 #

1. 模拟MLFlow的SQLAlchemy连接测试 #

进入MLFlow容器后，用Python直接测试SQLAlchemy连接（和MLFlow使用的连接方式一致）：

docker exec -it mlflow-server bash
pip install sqlalchemy pymysql  # 若容器内未安装
python << EOF
from sqlalchemy import create_engine
# 替换为实际的连接参数
engine = create_engine("mysql+pymysql://mlflow:你的密码@172.17.0.2:${MYSQL_PORT1}/mlflow")
try:
    connection = engine.connect()
    print("SQLAlchemy连接成功")
    connection.close()
except Exception as e:
    print(f"连接失败: {str(e)}")
EOF

2. 开启MLFlow调试日志 #

启动MLFlow时增加--log-level DEBUG参数，获取更详细的连接过程日志：

MLFLOW_CMD="mlflow server --backend-store-uri ${DB_BACKEND} --artifacts-destination /mlartifacts --serve-artifacts -h 0.0.0.0 -p 5000 --log-level DEBUG"

通过docker logs mlflow-server查看详细日志，排查连接失败的具体环节。

3. 检查MySQL的连接配置 #

• 进入MySQL容器，检查my.cnf中的bind-address设置是否为0.0.0.0（允许所有IP访问）：

  docker exec -it mysql-server cat /etc/mysql/my.cnf | grep bind-address
  

• 查看MySQL的用户权限，确认mlflow用户允许从MLFlow容器的IP或%（任意IP）访问：

  SELECT user, host FROM mysql.user WHERE user='mlflow';
  

  若权限不足，执行：

  GRANT ALL PRIVILEGES ON mlflow.* TO 'mlflow'@'%' IDENTIFIED BY '你的密码';
  FLUSH PRIVILEGES;
  

4. 检查Docker网络通信 #

• 查看两个容器的网络归属，确认在同一网络：

  docker inspect mlflow-server | grep Networks
  docker inspect mysql-server | grep Networks
  

• 在MLFlow容器内ping MySQL容器的IP或名称，确认网络连通性：

  docker exec mlflow-server ping 172.17.0.2
  docker exec mlflow-server ping mysql-server
  

内容的提问来源于stack exchange，提问作者user20452300