Python3.9下PyHive连接Hive在AWS Lambda报错：找不到sasl.saslwrapper

我有一个基于Python3.9、使用PyHive开发的小型程序，可通过Thrift服务器端点获取Hive数据库列表。程序在本地作为独立程序运行完全正常，但部署到AWS Lambda后，出现错误：No module named 'sasl.saslwrapper'。

连接代码片段 #

connection = hive.Connection(
        host='thrift server endpoint',
        port=10001,
        username='username',
        database='db name',
    )

本地依赖列表（与Lambda压缩包一致） #

drwxr-xr-x  3 root root  4096 Aug  1 15:09 google
drwxr-xr-x  2 root root  4096 Aug  1 15:09 protobuf-3.20.3.dist-info
drwxr-xr-x  2 root root  4096 Aug  1 15:10 mysql_connector_python-8.0.32.dist-info
drwxr-xr-x  5 root root  4096 Aug  1 15:10 mysql
drwxr-xr-x  5 root root  4096 Aug  1 15:10 mysqlx
-rw-r--r--  1 root root 34549 Aug  1 15:11 six.py
drwxr-xr-x  2 root root  4096 Aug  1 15:11 six-1.16.0.dist-info
drwxr-xr-x 10 root root  4096 Aug  1 15:11 future
drwxr-xr-x  2 root root  4096 Aug  1 15:11 future-0.18.3.dist-info
drwxr-xr-x  4 root root  4096 Aug  1 15:11 libpasteurize
drwxr-xr-x  4 root root  4096 Aug  1 15:11 libfuturize
drwxr-xr-x  7 root root  4096 Aug  1 15:12 past
drwxr-xr-x  6 root root  4096 Aug  1 15:12 dateutil
drwxr-xr-x  2 root root  4096 Aug  1 15:12 python_dateutil-2.8.2.dist-info
drwxr-xr-x  3 root root  4096 Aug  1 15:13 TCLIService
drwxr-xr-x  3 root root  4096 Aug  1 15:14 pyhive
drwxr-xr-x  2 root root  4096 Aug  1 15:14 pure_sasl-0.6.2.dist-info
drwxr-xr-x  3 root root  4096 Aug  1 15:14 puresasl
-rw-r--r--  1 root root   539 Aug  1 15:15 protobuf-3.20.3-py3.9-nspkg.pth
-rw-r--r--  1 root root   152 Aug  1 15:15 distutils-precedence.pth
drwxr-xr-x  7 root root  4096 Aug  1 15:16 setuptools
drwxr-xr-x  2 root root  4096 Aug  1 15:16 setuptools-58.1.0.dist-info
-rw-r--r--  1 root root  1916 Aug  1 15:25 hive_table_ddl.sql
drwxr-xr-x  2 root root  4096 Aug  1 16:59 requests-2.28.2.dist-info
drwxr-xr-x  2 root root  4096 Aug  1 16:59 requests
drwxr-xr-x  5 root root  4096 Aug  1 17:02 charset_normalizer
drwxr-xr-x  2 root root  4096 Aug  1 17:02 charset_normalizer-3.1.0.dist-info
drwxr-xr-x  3 root root  4096 Aug  1 17:05 idna
drwxr-xr-x  2 root root  4096 Aug  1 17:05 idna-3.4.dist-info
drwxr-xr-x  2 root root  4096 Aug  1 17:42 certifi-2023.7.22.dist-info
drwxr-xr-x  2 root root  4096 Aug  1 17:42 certifi
drwxr-xr-x  3 root root  4096 Aug  1 17:54 sasl
drwxr-xr-x  2 root root  4096 Aug  1 17:54 sasl-0.3.1.dist-info
drwxr-xr-x  6 root root  4096 Aug  1 18:03 thrift
drwxr-xr-x  2 root root  4096 Aug  1 18:03 thrift_sasl-0.4.3.dist-info
drwxr-xr-x  3 root root  4096 Aug  1 18:03 thrift_sasl
drwxr-xr-x  2 root root  4096 Aug  1 18:03 thrift-0.16.0.dist-info
drwxr-xr-x  2 root root  4096 Aug  1 18:19 __pycache__
-rw-r--r--  1 root root 10791 Aug  1 18:55 hive_update_compactor.py
-rw-r--r--  1 root root 11548 Aug  1 18:58 create_hive_queries.py

解决建议 #

• 改用纯Python版SASL库：你已安装pure_sasl，可以强制PyHive使用这个纯Python实现，避免依赖原生编译的sasl库。两种实现方式：
  1. 设置环境变量：

     import os
     os.environ['PYHIVE_USE_PURE_SASL'] = '1'
     

  2. 连接时直接指定sasl_class参数：

     from puresasl import SASLClient
     connection = hive.Connection(
             host='thrift server endpoint',
             port=10001,
             username='username',
             database='db name',
             sasl_class=SASLClient
         )
     

• 编译Lambda兼容的原生sasl库：如果必须使用原生sasl，需要在与Lambda相同的运行环境（Amazon Linux 2）下编译依赖。可以用Docker镜像来完成：

  docker run -v $(pwd):/app public.ecr.aws/sam/build-python3.9 bash -c "cd /app && pip install sasl==0.3.1 -t ."
  

  完成后将生成的依赖打包上传到Lambda即可。
• 检查架构兼容性：本地系统与Lambda的CPU架构可能不同（比如本地是arm64，Lambda默认是x86_64），导致编译的库无法兼容。需确保在对应架构的环境下安装依赖。

内容的提问来源于stack exchange，提问作者Ashish Kumar Mondal