Django Celery连接SSE-SQS加密SQS队列失败求助

问题背景 #

无法通过Django Celery连接启用SSE-SQS加密的Amazon SQS队列，连接非加密队列时一切正常。调整配置后出现两类错误，需修正配置解决。

当前配置 #

settings.py 初始配置 #

SQS_AWS_ACCESS_KEY_ID = 'xxxx'
SQS_AWS_SECRET_ACCESS_KEY = 'xxxx'
SQS_AWS_QUEUE_NAME = 'sqs.eu-west-2.amazonaws.com/xxxx/xxx-celery-broker'

broker_url = f"sqs://{SQS_AWS_ACCESS_KEY_ID}:{SQS_AWS_SECRET_ACCESS_KEY}@{SQS_AWS_QUEUE_NAME}"
CELERY_BROKER_URL = broker_url
CELERY_RESULT_BACKEND = None
CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP = True 

启动Worker时的错误 #

botocore.exceptions.ClientError: An error occurred (InvalidSecurity) when calling the GetQueueAttributes operation: All requests to this queue must use HTTPS and SigV4.

修改队列地址为HTTPS后的配置 #

SQS_AWS_QUEUE_NAME = 'https://sqs.eu-west-2.amazonaws.com/xxxx/xxx-celery-broker'

修改后出现的新错误 #

Cannot connect to sqs://xxx:**@https//sqs.eu-west-2.amazonaws.com/xxx/xxx-celery-broker: Could not connect to the endpoint URL: "http://https/".

celery.py 配置 #

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "proj.settings")

# Create a Celery instance and configure it to use SQS
app = Celery("proj")

# Load task modules from all registered Django app configs.
app.config_from_object("django.conf:settings", namespace="CELERY")

# Auto-discover tasks in all installed apps
app.autodiscover_tasks(settings.INSTALLED_APPS)
app.conf.task_default_queue = 'xxx-celery-broker'

调试日志片段 #

Making request for OperationModel(name=GetQueueAttributes) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.26.135 Python/3.11.6 Linux/6.2.0-1017-aws Botocore/1.29.165'}, 'body': {'Action': 'GetQueueAttributes', 'Version': '2012-11-05', 'QueueUrl': 'https://sqs.eu-west-2.amazonaws.com/id/name', 'AttributeName.1': 'ApproximateNumberOfMessages'}, 'url': 'http://https/', 'context': {'client_region': 'eu-west-2', 'client_config': <botocore.config.Config object at 0x7f4d28836b50>, 'has_streaming_input': False, 'auth_type': None}} 

解决方案 #

无需修改队列名格式，通过添加Celery broker传输选项强制使用HTTPS和SigV4签名即可解决问题：

修改settings.py，恢复队列名原始格式，并添加CELERY_BROKER_TRANSPORT_OPTIONS配置：

SQS_AWS_ACCESS_KEY_ID = 'xxxx'
SQS_AWS_SECRET_ACCESS_KEY = 'xxxx'
# 恢复为不带https://的队列地址格式
SQS_AWS_QUEUE_NAME = 'sqs.eu-west-2.amazonaws.com/xxxx/xxx-celery-broker'

broker_url = f"sqs://{SQS_AWS_ACCESS_KEY_ID}:{SQS_AWS_SECRET_ACCESS_KEY}@{SQS_AWS_QUEUE_NAME}"
CELERY_BROKER_URL = broker_url
CELERY_RESULT_BACKEND = None
CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP = True 

# 添加传输选项，强制HTTPS和SigV4签名
CELERY_BROKER_TRANSPORT_OPTIONS = {
    'region': 'eu-west-2',
    'use_ssl': True,
    'signature_version': 'v4'
}

原理说明 #

• 加密队列要求所有请求使用HTTPS和SigV4签名，Celery的SQS传输默认可能未启用这些配置，通过broker_transport_options显式指定后可满足要求。
• 直接在队列名前添加https://会导致broker_url解析错误，生成无效的http://https/端点，因此需保持队列名的原始格式。

内容的提问来源于stack exchange，提问作者stackoverflow1234