如何在AWS OpenSearch中注册稀疏编码预训练模型

问题场景 #

在AWS OpenSearch 2.11版本中，尝试部署预训练模型amazon/neural-sparse/opensearch-neural-sparse-encoding-v1以实现神经稀疏搜索，注册模型时遭遇失败。

提交的模型注册请求 #

POST /_plugins/_ml/models/_register
{
    "name": "amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1",
    "version": "1.0.0",
    "model_group_id": "<group-id>",
    "description": "This is a neural sparse encoding model: It transfers text into sparse vector, and then extract nonzero index and value to entry and weights. It serves only in ingestion and customer should use tokenizer model in query.",
    "model_format": "TORCH_SCRIPT",
    "function_name": "SPARSE_ENCODING",
    "model_content_hash_value": "9a41adb6c13cf49a7e3eff91aef62ed5035487a6eca99c996156d25be2800a9a",
    "url":  "https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1/1.0.0/torch_script/opensearch-neural-sparse-encoding-doc-v1-1.0.0-torch_script.zip"
}

首次错误响应 #

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "To upload custom model user needs to enable allow_registering_model_via_url settings. Otherwise please use opensearch pre-trained models."
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "To upload custom model user needs to enable allow_registering_model_via_url settings. Otherwise please use opensearch pre-trained models."
  },
  "status": 400
}

修改集群设置时的错误 #

为解决上述问题，尝试将allow_registering_model_via_url设置为true，提交请求：

PUT /_cluster/settings
{
    "persistent": {
        "plugins.ml_commons.allow_registering_model_via_url": true
    }
}

收到如下错误：

{
  "Message": "Your request: '/_cluster/settings' payload is not allowed."
}

解决方案 #

AWS托管OpenSearch服务限制了对allow_registering_model_via_url这类集群参数的直接修改，正确部署该模型的方式如下：

• 控制台可视化操作：通过AWS OpenSearch控制台进入ML Commons插件，选择「从模型仓库导入」，直接选取官方预训练的稀疏编码模型完成部署，无需手动构造URL注册请求。
• API/CLI方式：使用AWS提供的专用模型注册API或CLI命令，调用官方模型仓库的接口完成模型导入，避开自定义URL注册的限制。

该模型属于OpenSearch官方预训练稀疏编码模型范畴，无需通过自定义URL上传，直接利用官方提供的模型导入流程即可完成部署。

内容的提问来源于stack exchange，提问作者user23108725