将JSON文档导入Elasticsearch触发TransportError问题排查与解决

你在使用json.load加载JSON文件后写入Elasticsearch时遇到了TransportError，错误核心是mapper_parsing_exception下的not_x_content_exception，提示压缩器检测只能针对xcontent字节或压缩的xcontent字节。

原示例代码 #

inputFile = sys.argv[1]
with open(inputFile) as jsonFile:
    jsonData = json.load(jsonFile)
es = Elasticsearch([{'host':'x.x.x.175', 'port':9200}])
es.index(index="fmb_420", doc_type='fmb_420', body=jsonData)

错误日志 #

Traceback (most recent call last):
  File "FMB_DB.py", line 12, in <module>
    es.index(index="fmb_420", doc_type='fmb_420', body=jsonData)
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 319, in index
    _make_path(index, doc_type, id), params=params, body=body)
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 314, in perform_request
    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 180, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 125, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, u'mapper_parsing_exception', u'not_x_content_exception: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes')

问题根源与解决办法 #

这个错误的核心原因是你的JSON文件没有顶层根元素——Elasticsearch的index接口要求传入的body是结构完整的JSON对象（符合ES文档格式的结构），如果原JSON文件是无嵌套的零散键值对，就会导致ES无法正确解析内容。

你已经通过为JSON文件添加根元素解决了问题，举个直观的例子：

• 原错误的JSON内容（无根元素）：

  "key1": "value1",
  "key2": "value2"
  

• 修改后（添加顶层根对象）：

  {
    "key1": "value1",
    "key2": "value2"
  }
  

或者用一个自定义键包裹原内容，只要保证整个JSON有唯一的顶层结构，就能被ES正常解析。

内容的提问来源于stack exchange，提问作者Shankar