amd64架构下arrow_odbc与MySQL ODBC驱动libmyodbc9兼容问题求助

在amd64架构环境中，使用Polars通过ODBC驱动连接SQL Server和MySQL数据库时出现异常，Polars依赖的arrow_odbc组件会触发驱动文件找不到的错误。相同配置在arm64架构下可正常运行。

Dockerfile配置 #

FROM --platform=linux/amd64 python:3.12-slim AS build
# FROM --platform=linux/arm64 python:3.12-slim AS build

# install common dependencies
RUN apt update -y && apt install -y curl gnupg unixodbc unixodbc-dev wget

# install sqlserver odbc driver
RUN curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor -o /usr/share/keyrings/microsoft-prod.gpg && \
    curl https://packages.microsoft.com/config/debian/12/prod.list | tee /etc/apt/sources.list.d/mssql-release.list && \
    apt update -y && ACCEPT_EULA=Y apt install -y msodbcsql18

# install mysql odbc driver
ARG MYSQL_CONNECTOR_ODBC=mysql-connector-odbc-9.0.0-linux-glibc2.28-x86-64bit
# ARG MYSQL_CONNECTOR_ODBC=mysql-connector-odbc-9.0.0-linux-glibc2.28-aarch64

RUN wget -q https://dev.mysql.com/get/Downloads/Connector-ODBC/9.0/${MYSQL_CONNECTOR_ODBC}.tar.gz && \
    gunzip ${MYSQL_CONNECTOR_ODBC}.tar.gz && \
    tar xvf ${MYSQL_CONNECTOR_ODBC}.tar && \
    cp -r ${MYSQL_CONNECTOR_ODBC}/bin/* /usr/local/bin && \
    cp -r ${MYSQL_CONNECTOR_ODBC}/lib/* /usr/local/lib && \
    myodbc-installer -a -d -n "MySQL ODBC 9.0 Unicode Driver" -t "Driver=/usr/local/lib/libmyodbc9w.so"

# install python dependencies
RUN pip install -U pip pdm
ENV PYTHONPATH="${PYTHONPATH}:/app"

WORKDIR /app
COPY tasks/ingestion/[^task|Dockerfile]* ./
RUN pdm install --global --project .

COPY libraries/python/aws/ ./
COPY libraries/python/ingestion/ ./
COPY tasks/ingestion/task/ ./

CMD [ "python", "main.py" ]

测试情况 #

• 仅使用pyodbc连接MySQL可正常运行

import pyodbc

conn_str = "DRIVER={MySQL ODBC 9.0 Unicode Driver};SERVER=host.docker.internal;DATABASE=<db>;UID=<id>;PWD=<pwd>;ENCRYPT=yes;TrustServerCertificate=yes;"
stmt = "select * from <table> limit 500"

conn = pyodbc.connect(conn_str)
cursor = conn.execute(stmt)

• 先导入arrow_odbc再用pyodbc连接，或直接用arrow_odbc读取数据均失败
  失败脚本1：

  import pyodbc
  from arrow_odbc import read_arrow_batches_from_odbc
  
  conn_str = "DRIVER={MySQL ODBC 9.0 Unicode Driver};SERVER=host.docker.internal;DATABASE=<db>;UID=<id>;PWD=<pwd>;ENCRYPT=yes;TrustServerCertificate=yes;"
  stmt = "select * from <table> limit 500"
  
  conn = pyodbc.connect(conn_str)
  cursor = conn.execute(stmt)
  
  reader = read_arrow_batches_from_odbc(query=stmt, connection_string=conn_str)
  

  报错信息：

  Traceback (most recent call last):
    File "/app/test_odbc_ko.py", line 7, in <module>
      conn = pyodbc.connect(conn_str)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  pyodbc.Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib '/usr/local/lib/libmyodbc9w.so' : file not found (0) (SQLDriverConnect)")
  

  失败脚本2：

  from arrow_odbc import read_arrow_batches_from_odbc
  
  conn_str = "DRIVER={MySQL ODBC 9.0 Unicode Driver};SERVER=host.docker.internal;DATABASE=<db>;UID=<id>;PWD=<pwd>;ENCRYPT=yes;TrustServerCertificate=yes;"
  stmt = "select * from <table> limit 500"
  
  reader = read_arrow_batches_from_odbc(query=stmt, connection_string=conn_str)
  

  报错信息：

  Traceback (most recent call last):
    File "/app/test_odbc_ko.py", line 1, in <module>
    File "/usr/local/lib/python3.12/site-packages/arrow_odbc/reader.py", line 495, in read_arrow_batches_from_odbc
      reader.query(
    File "/usr/local/lib/python3.12/site-packages/arrow_odbc/reader.py", line 102, in query
      connection = connect_to_database(
                   ^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.12/site-packages/arrow_odbc/connect.py", line 59, in connect_to_database
      raise_on_error(error)
    File "/usr/local/lib/python3.12/site-packages/arrow_odbc/error.py", line 30, in raise_on_error
      raise Error(error_out)
  arrow_odbc.error.Error: ODBC emitted an error calling 'SQLDriverConnect':
  State: 01000, Native error: 0, Message: [unixODBC][Driver Manager]Can't open lib '/usr/local/lib/libmyodbc9w.so' : file not found
  

• 先通过pyodbc建立连接再导入arrow_odbc可正常运行

import pyodbc

conn_str = "DRIVER={MySQL ODBC 9.0 Unicode Driver};SERVER=host.docker.internal;DATABASE=<db>;UID=<id>;PWD=<pwd>;ENCRYPT=yes;TrustServerCertificate=yes;"
stmt = "select * from <table> limit 500"

conn = pyodbc.connect(conn_str)
cursor = conn.execute(stmt)

from arrow_odbc import read_arrow_batches_from_odbc

reader = read_arrow_batches_from_odbc(query=stmt, connection_string=conn_str)

已尝试的解决方法 #

• 更换Alpine、Ubuntu作为基础镜像
• 改用deb包安装MySQL连接器

以上操作均未解决问题，现寻求amd64架构下的兼容解决方案。

内容的提问来源于stack exchange，提问作者Lorenzo Sciarra