AWS OpenSearch Python客户端认证连接错误求助

问题背景 #

无法连接到AWS OpenSearch实例，环境变量已正确加载。尝试使用elasticsearch客户端时，因RequestsHttpConnection被移除导致该客户端不再支持AWS认证，转而使用opensearch-py库后出现认证错误。

代码示例 #

import os
from dotenv import load_dotenv
from opensearchpy import OpenSearch, helpers
from requests_aws4auth import AWS4Auth


load_dotenv()

host = os.environ.get("ES_HOST")
region = os.environ.get("ES_REGION")
service = os.environ.get("ES_SERVICE")
port = os.environ.get("ES_PORT")
index = os.environ.get("ES_INDEX")
access_key = os.environ.get("AWS_ACCESS_KEY_ID")
secret_key = os.environ.get("AWS_SECRET_ACCESS_KEY")

awsauth = AWS4Auth(access_key, secret_key, region, 'es')

es = OpenSearch(
    hosts=[{'host': host, 'port': port, 'scheme': 'https'}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    )

print(es.cluster.health())

报错信息 #

Traceback (most recent call last):
  File "/repo/.venv/lib/python3.9/site-packages/opensearchpy/connection/http_urllib3.py", line 276, in perform_request
    request_headers.update(self.http_auth(method, full_url, body))
TypeError: __call__() takes 2 positional arguments but 4 were given

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/repo/scratch/test_new.py", line 26, in <module>
    print(es.cluster.health())
  File "/repo/.venv/lib/python3.9/site-packages/opensearchpy/client/utils.py", line 176, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/repo/.venv/lib/python3.9/site-packages/opensearchpy/client/cluster.py", line 131, in health
    return self.transport.perform_request(
  File "/repo/.venv/lib/python3.9/site-packages/opensearchpy/transport.py", line 453, in perform_request
    raise e
  File "/repo/.venv/lib/python3.9/site-packages/opensearchpy/transport.py", line 416, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/repo/.venv/lib/python3.9/site-packages/opensearchpy/connection/http_urllib3.py", line 295, in perform_request
    raise ConnectionError("N/A", str(e), e)
opensearchpy.exceptions.ConnectionError: ConnectionError(__call__() takes 2 positional arguments but 4 were given) caused by: TypeError(__call__() takes 2 positional arguments but 4 were given)

环境信息 #

• Python版本：3.9.12
• 依赖包版本：
  • opensearch-py: 2.7.1
  • requests-aws4auth: 1.3.1
  • urllib3: 1.26.19

解决方案 #

问题根源是opensearch-py 2.x版本对http_auth可调用对象的参数要求，与requests-aws4auth.AWS4Auth的调用签名不匹配：opensearch-py会传入3个参数（method, url, body），但AWS4Auth的__call__方法仅接受2个参数（url, method），且顺序相反。

方法1：使用opensearch-py内置AWS认证类（推荐） #

opensearch-py 2.x版本内置了AWS认证支持，无需依赖第三方库，直接使用AWSV4SignerAuth即可：

import os
from dotenv import load_dotenv
from opensearchpy import OpenSearch, AWSV4SignerAuth
import boto3

load_dotenv()

host = os.environ.get("ES_HOST")
region = os.environ.get("ES_REGION")
port = os.environ.get("ES_PORT")

# 通过boto3自动获取AWS凭证
credentials = boto3.Session().get_credentials()
auth = AWSV4SignerAuth(credentials, region, 'es')

es = OpenSearch(
    hosts=[{'host': host, 'port': port, 'scheme': 'https'}],
    http_auth=auth,
    use_ssl=True,
    verify_certs=True,
)

print(es.cluster.health())

方法2：包装AWS4Auth适配调用签名 #

若必须使用requests-aws4auth，可自定义包装函数调整参数顺序与数量：

import os
from dotenv import load_dotenv
from opensearchpy import OpenSearch, helpers
from requests_aws4auth import AWS4Auth


load_dotenv()

host = os.environ.get("ES_HOST")
region = os.environ.get("ES_REGION")
service = os.environ.get("ES_SERVICE")
port = os.environ.get("ES_PORT")
index = os.environ.get("ES_INDEX")
access_key = os.environ.get("AWS_ACCESS_KEY_ID")
secret_key = os.environ.get("AWS_SECRET_ACCESS_KEY")

awsauth = AWS4Auth(access_key, secret_key, region, 'es')

# 包装函数适配opensearch-py的参数要求
def auth_wrapper(method, url, body=None):
    # 调整参数顺序，匹配AWS4Auth的调用格式
    return awsauth(url, method)

es = OpenSearch(
    hosts=[{'host': host, 'port': port, 'scheme': 'https'}],
    http_auth=auth_wrapper,
    use_ssl=True,
    verify_certs=True,
)

print(es.cluster.health())

额外检查项 #

• 确认AWS凭证拥有访问目标OpenSearch实例的权限（如AmazonOpenSearchServiceFullAccess或自定义权限策略）
• 检查OpenSearch实例的安全组规则，允许当前机器IP访问对应端口
• 验证ES_HOST环境变量值为实例域名，不带http://或https://前缀

内容的提问来源于stack exchange，提问作者tim_76