插入大量数据后Milvus抛出异常问题求助
批量插入海量数据时Milvus抛出SegmentID分配错误问题
- 批量插入约300万条数据(每次插入1万条,embedding维度为384)后,Milvus抛出如下错误;当embedding维度翻倍至768时,插入100-150万条数据后会出现相同或类似错误。
错误堆栈信息:
File "/ec/dgt/local/exodus/home/ewetzha/dev/enginefactory-dev/artefacts/venvs/venv-sentencetransformer/lib64/python3.6/site-packages/pymilvus/orm/collection.py", line 431, in insert timeout=timeout, schema=self._schema_dict, **kwargs) File "/ec/dgt/local/exodus/home/ewetzha/dev/enginefactory-dev/artefacts/venvs/venv-sentencetransformer/lib64/python3.6/site-packages/pymilvus/decorators.py", line 109, in handler raise e File "/ec/dgt/local/exodus/home/ewetzha/dev/enginefactory-dev/artefacts/venvs/venv-sentencetransformer/lib64/python3.6/site-packages/pymilvus/decorators.py", line 105, in handler return func(*args, **kwargs) File "/ec/dgt/local/exodus/home/ewetzha/dev/enginefactory-dev/artefacts/venvs/venv-sentencetransformer/lib64/python3.6/site-packages/pymilvus/decorators.py", line 136, in handler ret = func(self, *args, **kwargs) File "/ec/dgt/local/exodus/home/ewetzha/dev/enginefactory-dev/artefacts/venvs/venv-sentencetransformer/lib64/python3.6/site-packages/pymilvus/decorators.py", line 85, in handler raise e File "/ec/dgt/local/exodus/home/ewetzha/dev/enginefactory-dev/artefacts/venvs/venv-sentencetransformer/lib64/python3.6/site-packages/pymilvus/decorators.py", line 50, in handler return func(self, *args, **kwargs) File "/ec/dgt/local/exodus/home/ewetzha/dev/enginefactory-dev/artefacts/venvs/venv-sentencetransformer/lib64/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 399, in batch_insert raise err File "/ec/dgt/local/exodus/home/ewetzha/dev/enginefactory-dev/artefacts/venvs/venv-sentencetransformer/lib64/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 395, in batch_insert raise MilvusException(response.status.error_code, response.status.reason) pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error)>
- 目前唯一可行的恢复插入操作的方式是重启Milvus服务。
内容的提问来源于stack exchange,提问作者user22171862
相关产品推荐
相关产品推荐

