无法向Azure认知搜索索引加载大型数据集的求助

你的问题核心是SearchIndexingBufferedSender未按预期完成批量上传，仅成功写入少量文档，原因大概率是默认批量参数未适配场景，或上传过程中存在未捕获的异常导致批次中断。以下是针对性解决方法：

1. 调整批量参数并添加异常捕获 #

Azure认知搜索的批量API每批最多支持1000条文档，你可以显式设置batch_size参数匹配该限制，同时添加异常处理排查潜在错误，确保所有批次正常执行：

file = df.to_dict(orient="records")

try:
    with SearchIndexingBufferedSender(
        endpoint=service_endpoint,
        index_name=index_name,
        credential=credential,
        batch_size=1000,  # 匹配Azure搜索的批量上限
        max_retries=3     # 增加重试次数应对临时网络问题
    ) as batch_client:
        batch_client.upload_documents(documents=file)
        # 显式触发缓冲文档上传，确保所有批次提交完成
        batch_client.flush()
    print(f"Uploaded {len(file)} documents in total to {index_name}")
except Exception as e:
    print(f"Upload failed with error: {str(e)}")

2. 验证主键唯一性 #

检查DataFrame中对应搜索索引主键的字段是否存在重复值。若有重复主键，后续文档会覆盖之前的条目，可能造成仅第一条保留的假象。可通过以下代码验证：

# 替换"id"为你的实际主键字段名
if df["id"].duplicated().any():
    print("发现重复主键，请处理后再上传")
else:
    print("主键无重复")

3. 手动分批调试定位问题 #

如果问题仍存在，可手动拆分批次上传并打印状态，快速定位失败的批次：

from azure.core.exceptions import HttpResponseError

file = df.to_dict(orient="records")
batch_size = 1000

for i in range(0, len(file), batch_size):
    batch = file[i:i+batch_size]
    try:
        with SearchIndexingBufferedSender(
            endpoint=service_endpoint,
            index_name=index_name,
            credential=credential,
            batch_size=batch_size
        ) as batch_client:
            batch_client.upload_documents(documents=batch)
            batch_client.flush()
        print(f"Successfully uploaded batch {i//batch_size + 1}, {len(batch)} documents")
    except HttpResponseError as e:
        print(f"Batch {i//batch_size + 1} failed: {e.message}")

内容的提问来源于stack exchange，提问作者Irina