Deepset Haystack连接带安全认证的Elasticsearch遇连接错误

问题描述 #

Docker部署了带安全认证的Elasticsearch，使用Haystack创建ElasticsearchDocumentStore时出现连接错误。已知需要通过HTTPS连接，但未找到Haystack的对应配置方法，已尝试9300和9200端口。

原代码示例 #

document_store = ElasticsearchDocumentStore(
    host='https://localhost', 
    username= 'elastic',
    password='{PASSWORD}',
    port=9200,
    index='squad_docs'
)

错误信息 #

• Python控制台错误：

ConnectionError: ConnectionError(('连接中止。', RemoteDisconnected('远程端无响应关闭连接'))) caused by: ProtocolError(('连接中止。', RemoteDisconnected('远程端无响应关闭连接')))

• Elasticsearch日志警告：

{"@timestamp":"2022-11-08T14:01:54.875Z", "log.level": "WARN", "message":"在HTTPS通道上收到明文HTTP流量，关闭连接 Netty4HttpChannel{localAddress=/172.18.0.2:9200, remoteAddress=/172.18.0.1:56568}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[c759978b5b07][transport_worker][T#1]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"vDXCcs4uQM-gcuf5vciZkA","elasticsearch.node.id":"VunPyrAPRbG9PrLFyZ1E8w","elasticsearch.node.name":"c759978b5b07","elasticsearch.cluster.name":"docker-cluster"}

解决方案 #

Haystack的ElasticsearchDocumentStore底层依赖官方elasticsearch-py客户端，只需通过scheme参数明确指定HTTPS协议，同时处理证书验证问题即可：

修改后的代码（测试环境） #

document_store = ElasticsearchDocumentStore(
    host='localhost', 
    username='elastic',
    password='{PASSWORD}',
    port=9200,
    index='squad_docs',
    scheme='https',
    verify_certs=False  # 测试环境可临时关闭证书验证，生产环境不建议
)

生产环境配置 #

生产环境需保留证书验证，将Docker中Elasticsearch容器内的/config/certs/http_ca.crt证书文件复制到本地，然后指定证书路径：

document_store = ElasticsearchDocumentStore(
    host='localhost', 
    username='elastic',
    password='{PASSWORD}',
    port=9200,
    index='squad_docs',
    scheme='https',
    ca_certs="/本地路径/http_ca.crt"
)

错误原因 #

原代码将https://写入host参数，但Haystack的host仅需填写主机名，协议需通过scheme单独指定。这导致客户端仍以HTTP协议发送请求到Elasticsearch的HTTPS端口，触发ES日志中“明文HTTP流量进入HTTPS通道”的警告，最终被ES断开连接。

内容的提问来源于stack exchange，提问作者eboraks