上传大文件至Azure Cognitive Search遇ServiceRequestError问题求助

问题场景 #

尝试将包含embeddings的索引推送到Azure Cognitive Search，使用以下代码上传：

# Upload some documents to the index
with open('index.json', 'r') as file:  
    documents = json.load(file)  
search_client = SearchClient(endpoint=service_endpoint, index_name=index_name, credential=credential)
result = search_client.upload_documents(documents, timeout = 50)  
print(f"Uploaded {len(documents)} documents") 

小文件可成功推送，但处理69MB的index.json时失败，报错信息：

ServiceRequestError                       Traceback (most recent call last)
Cell In[21], line 5
      3     documents = json.load(file)  
      4 search_client = SearchClient(endpoint=service_endpoint, index_name=index_name, credential=credential)
----> 5 result = search_client.upload_documents(documents, timeout = 50)  
      6 print(f"Uploaded {len(documents)} documents") 

File /usr/local/lib/python3.11/site-packages/azure/search/documents/_search_client.py:543, in SearchClient.upload_documents(self, documents, **kwargs)
    540 batch.add_upload_actions(documents)
    542 kwargs["headers"] = self._merge_client_headers(kwargs.get("headers"))
--> 543 results = self.index_documents(batch, **kwargs)
    544 return cast(List[IndexingResult], results)

File /usr/local/lib/python3.11/site-packages/azure/core/tracing/decorator.py:78, in distributed_trace..decorator..wrapper_use_tracer(*args, **kwargs)
     76 span_impl_type = settings.tracing_implementation()
     77 if span_impl_type is None:
--> 78     return func(*args, **kwargs)
     80 # Merge span is parameter is set, but only if no explicit parent are passed
     81 if merge_span and not passed_in_parent:

File /usr/local/lib/python3.11/site-packages/azure/search/documents/_search_client.py:641, in SearchClient.index_documents(self, batch, **kwargs)
    631 @distributed_trace
    632 def index_documents(self, batch: IndexDocumentsBatch, **kwargs: Any) -> List[IndexingResult]:
    633     """Specify a document operations to perform as a batch.
...
--> 381     raise error
    382 if _is_rest(request):
    383     from azure.core.rest._requests_basic import RestRequestsTransportResponse

ServiceRequestError: EOF occurred in violation of protocol (_ssl.c:2427)

错误原因 #

Azure Cognitive Search的批量索引接口有严格限制：单批请求最多包含1000个文档，或请求体大小不超过16MB（取两者中更小的限制）。69MB的文件远超该限制，导致服务端提前终止连接，触发SSL协议层面的EOF错误。

解决方案 #

1. 拆分文档为小批次上传 #

将大文档列表拆分为多个符合限制的小批次，逐个上传。示例代码：

import json
from azure.search.documents import SearchClient
from azure.core.credentials import AzureKeyCredential

# 配置你的搜索服务信息
service_endpoint = "你的搜索服务端点"
index_name = "你的索引名称"
credential = AzureKeyCredential("你的API密钥")

# 加载全部文档
with open('index.json', 'r') as file:  
    documents = json.load(file)  

search_client = SearchClient(endpoint=service_endpoint, index_name=index_name, credential=credential)

# 设置批次大小（建议不超过500，避免接近16MB限制）
batch_size = 500
total_success = 0

# 分批上传
for batch_idx in range(0, len(documents), batch_size):
    current_batch = documents[batch_idx:batch_idx + batch_size]
    try:
        # 延长超时时间至120秒，适配大批次传输
        results = search_client.upload_documents(current_batch, timeout=120)
        # 统计成功数量
        success_num = sum(1 for res in results if res.succeeded)
        total_success += success_num
        print(f"批次 {batch_idx//batch_size + 1} 完成：成功上传 {success_num}/{len(current_batch)} 个文档")
    except Exception as e:
        print(f"批次 {batch_idx//batch_size + 1} 失败：{str(e)}")

print(f"全部上传完成，累计成功上传 {total_success}/{len(documents)} 个文档")

2. 调整超时时间 #

大批次上传需要更长的网络传输时间，原代码的50秒超时可能不足以完成请求，建议调整为120秒或更长（根据实际网络环境调整）。

3. 检查单个文档大小（可选） #

如果单个文档本身大小超过16MB，需要使用Azure Cognitive Search的分块上传功能，针对单个大文档进行拆分上传。但你的场景中主要是批量总大小超标，优先使用分批上传即可。

内容的提问来源于stack exchange，提问作者Herwini