本地用BiosentVec向Elasticsearch索引数据遇TimeoutError求助

问题场景 #

本地向Elasticsearch索引30万条观测数据，调用BiosentVec API生成嵌入向量时程序反复中断，出现连接超时错误：

requests.exceptions.ConnectionError: ('Connection aborted.', TimeoutError(10060, 'A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond', None, 10060, None))

完整报错栈：

File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\Anaconda3\envs\bsv_Env\lib\http\client.py", line 1348, in getresponse
    response.begin()
  File "C:\Users\Anaconda3\envs\bsv_Env\lib\http\client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "C:\Users\Anaconda3\envs\bsv_Env\lib\http\client.py", line 277, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "C:\Users\Anaconda3\envs\bsv_Env\lib\socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "C:\Users\Anaconda3\envs\bsv_Env\lib\ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "C:\Users\Anaconda3\envs\bsv_Env\lib\ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\AppData\Roaming\Python\Python38\site-packages\requests\adapters.py", line 489, in send
    resp = conn.urlopen(
  File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\util\retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\packages\six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "C:\Users\AppData\Roaming\Python\Python38\site-packages\urllib3\connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\Anaconda3\envs\bsv_Env\lib\http\client.py", line 1348, in getresponse
    response.begin()
  File "C:\Users\Anaconda3\envs\bsv_Env\lib\http\client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "C:\Users\Anaconda3\envs\bsv_Env\lib\http\client.py", line 277, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "C:\Users\Anaconda3\envs\bsv_Env\lib\socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "C:\Users\Anaconda3\envs\bsv_Env\lib\ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "C:\Users\Anaconda3\envs\bsv_Env\lib\ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
urllib3.exceptions.ProtocolError: ('Connection aborted.', TimeoutError(10060, 'A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond', None, 10060, None))

当前使用的代码：

client = Elasticsearch(hosts=const.HOSTS, timeout=30, max_retries=10, retry_on_timeout=True)

def run_bio_sent_vec_api(input):
    result = {} 
    script_query = {
      "query": input
    }
    headers = {"Accept": "application/json", "Content-Type": "application/json",
               'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
    url = "https://biosentvec-proxy-r7tozhc6ua-uc.a.run.app/fetchEmbedding"
    response = requests.post(url, headers=headers, json=script_query)
    if response.status_code == 200:
        result = response.json()['biosentvec']
    else:
        print('Error:', response.status_code)
    return result

解决方案 #

1. 给请求添加超时参数 #

当前requests.post未设置超时，一旦API响应缓慢就会触发系统级超时。添加timeout参数，分别配置连接超时和读取超时：

# 连接超时10秒，读取超时30秒
response = requests.post(url, headers=headers, json=script_query, timeout=(10, 30))

2. 实现请求重试机制 #

使用urllib3的重试策略，针对连接错误、超时等异常自动重试，避免单次失败导致程序中断：

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

def run_bio_sent_vec_api(input):
    result = {} 
    script_query = {
      "query": input
    }
    headers = {"Accept": "application/json", "Content-Type": "application/json",
               'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
    url = "https://biosentvec-proxy-r7tozhc6ua-uc.a.run.app/fetchEmbedding"
    
    # 配置重试策略
    retry_strategy = Retry(
        total=5,  # 总重试次数
        backoff_factor=1,  # 重试间隔时间（1, 2, 4...秒）
        status_forcelist=[429, 500, 502, 503, 504],  # 触发重试的状态码
        allowed_methods=["POST"]  # 允许重试的请求方法
    )
    adapter = HTTPAdapter(max_retries=retry_strategy)
    session = requests.Session()
    session.mount("https://", adapter)
    
    try:
        response = session.post(url, headers=headers, json=script_query, timeout=(10, 30))
        if response.status_code == 200:
            result = response.json()['biosentvec']
        else:
            print('Error:', response.status_code)
    except requests.exceptions.RequestException as e:
        print(f"请求异常: {e}")
    return result

3. 批量处理请求 #

单条调用API处理30万条数据会产生大量请求，容易触发API限流或网络拥堵。如果BiosentVec API支持批量请求，修改代码批量提交数据：

# 假设API接受批量query参数
def run_bio_sent_vec_batch(input_list):
    result = []
    script_query = {
      "queries": input_list
    }
    # 后续请求逻辑类似，批量获取嵌入向量

4. 控制请求频率 #

添加短延迟避免频繁请求，防止被API服务器限制：

import time

# 在每次请求后添加0.1秒延迟
time.sleep(0.1)

5. 排查网络环境 #

• 直接在浏览器或命令行测试API连通性：curl -X POST "https://biosentvec-proxy-r7tozhc6ua-uc.a.run.app/fetchEmbedding" -H "Content-Type: application/json" -d '{"query": "test"}'
• 检查本地防火墙、代理设置，确认没有拦截请求
• 测试网络稳定性，避免因网络波动导致连接中断

内容的提问来源于stack exchange，提问作者cndze