在AWS Lambda使用xarray-zonal遇RuntimeError缓存函数错误求助

本地运行正常的xarray-spatial分区统计代码，部署到AWS Lambda时抛出**RuntimeError: cannot cache function 'extract_scaled'**错误。代码从xarray数据集提取数据，调用zonal_stats本地可生成pandas DataFrame，但Lambda环境下执行失败。

相关业务代码 #

def zonal(self, data_id, zonal_id):
        zs = zonal_stats(self.dataset[zonal_id], self.dataset[data_id])
        pixel_area = round(math.pow(self.dataset[data_id].attrs['pixel_length'],2),2)
        zs['area_ha'] = pixel_area * zs['count'] / 10000
        return zs

完整报错栈 #

[ERROR] RuntimeError: cannot cache function 'extract_scaled': no locator available for file '/var/task/datashader/composite.py'
Traceback (most recent call last):
  File "/var/lang/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/var/task/app.py", line 2, in <module>
    from data import Data
  File "/var/task/data.py", line 14, in <module>
    from xrspatial import zonal_stats
  File "/var/task/xrspatial/__init__.py", line 3, in <module>
    from xrspatial.aspect import aspect  # noqa
  File "/var/task/xrspatial/aspect.py", line 10, in <module>
    from xrspatial.utils import ArrayTypeFunctionMapping, cuda_args, ngjit, not_implemented_func
  File "/var/task/xrspatial/utils.py", line 4, in <module>
    import datashader as ds
  File "/var/task/datashader/__init__.py", line 8, in <module>
    from .core import Canvas                                 # noqa (API import)
  File "/var/task/datashader/core.py", line 18, in <module>
    from . import reductions as rd
  File "/var/task/datashader/reductions.py", line 13, in <module>
    from datashader.transfer_functions._cuda_utils import (cuda_atomic_nanmin,
  File "/var/task/datashader/transfer_functions/__init__.py", line 16, in <module>
    from datashader.composite import composite_op_lookup, over, validate_operator
  File "/var/task/datashader/composite.py", line 30, in <module>
    def extract_scaled(x):
  File "/var/task/numba/core/decorators.py", line 212, in wrapper
    disp.enable_caching()
  File "/var/task/numba/core/dispatcher.py", line 863, in enable_caching
    self._cache = FunctionCache(self.py_func)
  File "/var/task/numba/core/caching.py", line 613, in __init__
    self._impl = self._impl_class(py_func)
  File "/var/task/numba/core/caching.py", line 350, in __init__
    raise RuntimeError("cannot cache function %r: no locator available "

这个错误根源是Numba在AWS Lambda的只读文件系统环境下无法缓存编译后的函数，以下是几种可行的解决办法：

• 方法1：调整Numba缓存路径或禁用缓存
  在Lambda代码的最顶部添加环境变量配置，让Numba使用Lambda可写的/tmp目录存储缓存：

  import os
  # 设置缓存目录为可写的/tmp
  os.environ['NUMBA_CACHE_DIR'] = '/tmp'
  # 或者直接禁用缓存（会牺牲部分性能）
  # os.environ['NUMBA_DISABLE_CACHE'] = '1'
  

• 方法2：使用AWS Lambda层部署依赖
  将xarray-spatial、datashader、numba等依赖打包成Lambda层，避免直接打包到代码包中导致的路径问题。打包命令示例：

  mkdir -p layer/python
  pip install --target ./layer/python xarray-spatial datashader numba
  cd layer && zip -r layer.zip python
  

  然后在Lambda控制台中将该层关联到函数。

• 方法3：优化代码包打包方式
  如果直接打包代码和依赖，确保打包时保留依赖文件的原始目录结构，不要使用会修改文件路径的压缩工具或打包插件。例如避免使用serverless框架的package.individually等可能破坏路径的选项。

• 方法4：升级依赖版本
  部分旧版本的numba、datashader在Lambda环境下存在文件定位的bug，升级到最新稳定版可解决问题。建议将numba升级到0.57.x以上，datashader升级到0.15.x以上。

内容的提问来源于stack exchange，提问作者Jamie Dunbar