在AWS Lambda中通过Python下载NLTK数据遇只读文件系统错误求助

问题描述 #

在AWS Lambda环境运行Python代码下载NLTK数据时，出现只读文件系统错误：

{"errorMessage": "[Errno 30] Read-only file system: 'layers'","errorType": "OSError","requestId": "","stackTrace": ["  File \"/var/lang/lib/python3.9/importlib/__init__.py\", line 127, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n","  File \"<frozen importlib._bootstrap>\", line 1030, in _gcd_import\n","  File \"<frozen importlib._bootstrap>\", line 1007, in _find_and_load\n","  File \"<frozen importlib._bootstrap>\", line 986, in _find_and_load_unlocked\n","  File \"<frozen importlib._bootstrap>\", line 680, in _load_unlocked\n","  File \"<frozen importlib._bootstrap_external>\", line 850, in exec_module\n","  File \"<frozen importlib._bootstrap>\", line 228, in _call_with_frames_removed\n","  File \"/opt/python/lib/python3.9/site-packages/datadog_lambda/handler.py\", line 30, in <module>\n    handler_module = import_module(modified_mod_name)\n","  File \"/var/lang/lib/python3.9/importlib/__init__.py\", line 127, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n","  File \"<frozen importlib._bootstrap>\", line 1030, in _gcd_import\n","  File \"<frozen importlib._bootstrap>\", line 1007, in _find_and_load\n","  File \"<frozen importlib._bootstrap>\", line 972, in _find_and_load_unlocked\n","  File \"<frozen importlib._bootstrap>\", line 228, in _call_with_frames_removed\n","  File \"<frozen importlib._bootstrap>\", line 1030, in _gcd_import\n","  File \"<frozen importlib._bootstrap>\", line 1007, in _find_and_load\n","  File \"<frozen importlib._bootstrap>\", line 986, in _find_and_load_unlocked\n","  File \"<frozen importlib._bootstrap>\", line 680, in _load_unlocked\n","  File \"<frozen importlib._bootstrap_external>\", line 850, in exec_module\n","  File \"<frozen importlib._bootstrap>\", line 228, in _call_with_frames_removed\n","  File \"/var/task/entrypoints/__init__.py\", line 14, in <module>\n    from utils.container import state_machine\n","  File \"/var/task/utils/container.py\", line 10, in <module>\n    from service_layer.models.nltk_downloader import is_nltk_data_download\n","  File \"/var/task/service_layer/models/nltk_downloader.py\", line 12, in <module>\n    nltk.download(\"punkt\", download_dir=environ[\"LAMBDA_NLTK_DATA\"])\n","  File \"/var/task/nltk/downloader.py\", line 777, in download\n    for msg in self.incr_download(info_or_id, download_dir, force):\n","  File \"/var/task/nltk/downloader.py\", line 642, in incr_download\n    yield from self._download_package(info, download_dir, force)\n","  File \"/var/task/nltk/downloader.py\", line 699, in _download_package\n    os.makedirs(download_dir)\n","  File \"/var/lang/lib/python3.9/os.py\", line 215, in makedirs\n    makedirs(head, exist_ok=exist_ok)\n","  File \"/var/lang/lib/python3.9/os.py\", line 225, in makedirs\n    mkdir(name, mode)\n"]}

相关代码：

nltk.download("punkt", download_dir=download_dir)
nltk.download("wordnet", download_dir=download_dir)
nltk.download("omw-1.4", download_dir=download_dir)

尝试过的路径如/layers/nltk_data、nltk_data均无效。

解决思路 #

1. 使用Lambda唯一可写目录/tmp #

AWS Lambda只有/tmp目录具备读写权限，需将下载路径指向该目录下的子文件夹，并告知NLTK数据位置：

import nltk
import os

# 设置可写的下载目录
download_dir = "/tmp/nltk_data"
# 确保目录存在，避免创建失败
os.makedirs(download_dir, exist_ok=True)

# 下载NLTK数据
nltk.download("punkt", download_dir=download_dir)
nltk.download("wordnet", download_dir=download_dir)
nltk.download("omw-1.4", download_dir=download_dir)

# 将数据目录添加到NLTK的搜索路径
nltk.data.path.append(download_dir)

2. 预先打包NLTK数据到Lambda层（推荐方案） #

每次函数启动时下载数据会增加冷启动时间，预先将NLTK数据打包成Lambda层是更优选择：

• 本地创建对应Python版本的目录结构：python/lib/pythonX.X/site-packages/nltk_data（X.X替换为Lambda使用的Python版本，如3.9）
• 本地运行命令下载数据到上述目录：

  nltk.download(["punkt", "wordnet", "omw-1.4"], download_dir="python/lib/python3.9/site-packages/nltk_data")
  

• 将python目录打包为zip文件
• 在AWS控制台创建Lambda层，上传该zip包
• 将层关联到目标Lambda函数
• 代码中无需再执行下载操作，直接使用NLTK即可，层中的数据会被自动识别

3. 检查环境变量配置 #

如果使用LAMBDA_NLTK_DATA环境变量指定路径，确保其值为/tmp/nltk_data，禁止指向/layers等只读目录。

内容的提问来源于stack exchange，提问作者Sazzad