基于AWS Lambda部署Mask-RCNN模型推理时遇OSError问题求助

问题背景 #

• 基于Mask-RCNN训练完成图像分割模型，计划部署到AWS Lambda实现推理服务
• 未采用EFS挂载模型文件，而是将模型放在项目内的maskrcnn目录，通过Dockerfile的COPY maskrcnn/ ./maskrcnn指令完成文件复制
• 本地测试调用完全正常，但部署后通过API Gateway端点触发推理时出现报错

项目结构 #

.
├── Dockerfile
├── __init__.py
├── app.py
├── requirements.txt
└── maskrcnn
    ├── config.py
    ├── __init__.py
    ├── m_rcnn.py
    ├── visualize.py
    ├── mask_rcnn_coco.h5
    ├── mask_rcnn_object_0005.h5
    └── model.py

报错信息 #

[ERROR] OSError: [Errno 38] Function not implemented
Traceback (most recent call last):
  File "/var/task/app.py", line 32, in lambda_handler
    r = test_model.detect([image])[0]
  File "/var/task/maskrcnn/model.py", line 2545, in detect
    self.keras_model.predict([molded_images, image_metas, anchors], verbose=0)
  File "/var/lang/lib/python3.8/site-packages/tensorflow/python/keras/engine/training_v1.py", line 988, in predict
    return func.predict(
  File "/var/lang/lib/python3.8/site-packages/tensorflow/python/keras/engine/training_arrays_v1.py", line 703, in predict
    return predict_loop(
  File "/var/lang/lib/python3.8/site-packages/tensorflow/python/keras/engine/training_arrays_v1.py", line 386, in model_iteration
    aggregator.create(batch_outs)
  File "/var/lang/lib/python3.8/site-packages/tensorflow/python/keras/engine/training_utils_v1.py", line 446, in create
    SliceAggregator(self.num_samples, self.batch_size)))
  File "/var/lang/lib/python3.8/site-packages/tensorflow/python/keras/engine/training_utils_v1.py", line 355, in __init__
    self._pool = get_copy_pool()
  File "/var/lang/lib/python3.8/site-packages/tensorflow/python/keras/engine/training_utils_v1.py", line 323, in get_copy_pool
    _COPY_POOL = multiprocessing.pool.ThreadPool(_COPY_THREADS)
  File "/var/lang/lib/python3.8/multiprocessing/pool.py", line 925, in __init__
    Pool.__init__(self, processes, initializer, initargs)
  File "/var/lang/lib/python3.8/multiprocessing/pool.py", line 196, in __init__
    self._change_notifier = self._ctx.SimpleQueue()
  File "/var/lang/lib/python3.8/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/var/lang/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
    self._rlock = ctx.Lock()
  File "/var/lang/lib/python3.8/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/var/lang/lib/python3.8/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/var/lang/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(

问题原因 #

AWS Lambda运行环境不支持Python multiprocessing模块中的进程同步原语（如SemLock），而TensorFlow/Keras在默认的predict流程中会启用多线程池处理数据复制，触发了Lambda不支持的系统调用。

解决方法 #

1. 禁用TensorFlow多线程配置 #

在加载模型前添加以下代码，强制TensorFlow使用单线程运行，避免触发多线程池：

import tensorflow as tf
import os

# 限制TensorFlow并行线程数
tf.config.threading.set_inter_op_parallelism_threads(1)
tf.config.threading.set_intra_op_parallelism_threads(1)

# 禁用多进程相关环境变量
os.environ['OMP_NUM_THREADS'] = '1'
os.environ['CUDA_VISIBLE_DEVICES'] = ''

2. 修改Mask-RCNN的detect方法 #

直接修改maskrcnn/model.py中的detect方法，在调用predict时显式关闭多进程：

# 原代码
# self.keras_model.predict([molded_images, image_metas, anchors], verbose=0)
# 修改后
self.keras_model.predict([molded_images, image_metas, anchors], verbose=0, use_multiprocessing=False)

3. 转换模型为SavedModel格式 #

将训练好的Mask-RCNN模型导出为TensorFlow SavedModel格式，使用TensorFlow原生方式加载模型，避免Keras旧版本的多线程兼容问题：

# 导出模型示例（训练完成后执行）
model.keras_model.save('maskrcnn_savedmodel')

# Lambda中加载模型
import tensorflow as tf
model = tf.keras.models.load_model('maskrcnn_savedmodel')

内容的提问来源于stack exchange，提问作者BodeTech