Bitbucket Pipelines中Elasticsearch连接失败问题求助

问题背景 #

Elasticsearch在本地开发环境、Elastic托管SaaS平台（预发布/生产）均可正常运行，但在Bitbucket Pipelines中配置后，无法连接到Elasticsearch实例，两种配置方案均触发连接错误。

已尝试的配置方案 #

方案一：测试步骤中直接启动Elasticsearch容器 #

- step: &unit_tests
    image: python:3.9
    name: Testing
    caches:
      - pip
    script:
      - pip install -r requirements.txt
      - export GC_API_DB_PORT=5432
      - export CS_DEPLOYMENT=development
      - docker run -d -p 9200:9200 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:8.6.2
      - python manage.py search_index --rebuild -f
      - python -m pytest --cov-report term --cov=gc_api --cov=gc_api_auth --cov-fail-under=85
    services:
      - postgres
      - docker

方案二：将Elasticsearch作为独立服务配置 #

elasticsearch:
  image: docker.elastic.co/elasticsearch/elasticsearch:8.6.2
  container_name: elasticsearch
  cap_add:
    - IPC_LOCK
  volumes:
    - elasticsearch-data:/usr/share/elasticsearch/data
  ports:
    - 9200:9200
    - 9300:9300
....
- step: &unit_tests
    image: python:3.9
    name: Testing
    caches:
      - pip
    script:
      - pip install -r requirements.txt
      - export GC_API_DB_PORT=5432
      - export ELASTIC_SEARCH_HOST=$BITBUCKET_DOCKER_HOST_INTERNAL
      - python -m pytest --cov-report term --cov=gc_api --cov=gc_api_auth --cov-fail-under=85
    services:
      - postgres
      - elasticsearch

错误栈信息 #

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/usr/local/lib/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.9/http/client.py", line 289, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/connection/http_requests.py", line 166, in perform_request
    response = self.session.send(prepared_request, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/atlassian/pipelines/agent/build/manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.9/site-packages/django_elasticsearch_dsl/management/commands/search_index.py", line 301, in handle
    for index in self.es_conn.indices.get_alias().values():
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/client/indices.py", line 642, in get_alias
    return self.transport.perform_request(
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/transport.py", line 417, in perform_request
    self._do_verify_elasticsearch(headers=headers, timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/transport.py", line 606, in _do_verify_elasticsearch
    raise error
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/transport.py", line 569, in _do_verify_elasticsearch
    _, info_headers, info_response = conn.perform_request(
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/connection/http_requests.py", line 194, in perform_request
    raise ConnectionError("N/A", str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))) caused by: ConnectionError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))

排查与解决建议 #

1. 解决Elasticsearch启动延迟问题 #

方案一中启动容器后直接执行索引重建，此时Elasticsearch尚未完成初始化，需要添加等待逻辑，确保服务就绪后再执行后续命令：

# 在docker run之后添加以下脚本
until curl -s http://localhost:9200/ > /dev/null; do
    echo "Waiting for Elasticsearch to start..."
    sleep 5
done

2. 禁用Elasticsearch 8.x默认安全配置 #

ES 8.x版本默认开启HTTPS和身份验证，本地环境可能已禁用，但容器镜像默认启用，导致连接失败。需在启动命令中添加环境变量关闭安全设置：

• 方案一修改docker run命令：

docker run -d -p 9200:9200 -e "discovery.type=single-node" -e "xpack.security.enabled=false" -e "xpack.security.enrollment.enabled=false" docker.elastic.co/elasticsearch/elasticsearch:8.6.2

• 方案二修改elasticsearch服务配置：

elasticsearch:
  image: docker.elastic.co/elasticsearch/elasticsearch:8.6.2
  container_name: elasticsearch
  cap_add:
    - IPC_LOCK
  environment:
    - discovery.type=single-node
    - xpack.security.enabled=false
    - xpack.security.enrollment.enabled=false
    - ES_JAVA_OPTS=-Xms512m -Xmx512m
  volumes:
    - elasticsearch-data:/usr/share/elasticsearch/data
  ports:
    - 9200:9200
    - 9300:9300

3. 修正方案二的连接地址 #

Bitbucket Pipelines中，服务容器之间通过服务名称直接访问，无需使用$BITBUCKET_DOCKER_HOST_INTERNAL，将环境变量改为：

export ELASTIC_SEARCH_HOST=elasticsearch

4. 配置Elasticsearch内存限制 #

Bitbucket Pipelines默认资源有限，ES需要足够内存才能正常启动，添加ES_JAVA_OPTS设置JVM内存（如上述方案二配置中的-Xms512m -Xmx512m），避免因内存不足导致服务启动失败。

内容的提问来源于stack exchange，提问作者David Sigley