Locust压测因Greenlet/Gevent错误无法停止，卡在CleanUp阶段

问题现象 #

在Google Cloud Platform上使用Locust对FastAPI接口进行性能测试，点击「停止」按钮时，会因Greenlet/Gevent相关错误间歇性无法停止测试，Locust卡在CleanUp阶段，请求仍持续发送。
该错误在更高并发峰值用户数和生成速率下触发概率更高，设置300峰值用户、100生成速率时几乎必然出现。此外未在on_stop()中执行任何操作。

错误日志 #

DEFAULT 2024-05-09T16:01:22.594432Z Exception ignored in: <finalize object at 0x3ef2919c3b20; dead>
ERROR 2024-05-09T16:01:22.595503Z Traceback (most recent call last): File "/usr/local/lib/python3.11/weakref.py", line 590, in __call__ return info.func(*info.args, **(info.kwargs or {}))
DEFAULT 2024-05-09T16:01:22.595511Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-05-09T16:01:22.595517Z File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1133, in _close_pool_connections
DEFAULT 2024-05-09T16:01:22.597778Z conn = pool.get(block=False)
DEFAULT 2024-05-09T16:01:22.597844Z ^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-05-09T16:01:22.597855Z File "/usr/local/lib/python3.11/queue.py", line 182, in get
DEFAULT 2024-05-09T16:01:22.598757Z self.not_full.notify()
DEFAULT 2024-05-09T16:01:22.598854Z File "/usr/local/lib/python3.11/threading.py", line 369, in notify
DEFAULT 2024-05-09T16:01:22.600847Z if not self._is_owned():
DEFAULT 2024-05-09T16:01:22.600879Z ^^^^^^^^^^^^^^^^
DEFAULT 2024-05-09T16:01:22.600883Z File "/usr/local/lib/python3.11/threading.py", line 282, in _is_owned
DEFAULT 2024-05-09T16:01:22.601021Z if self._lock.acquire(False):
DEFAULT 2024-05-09T16:01:22.601056Z ^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-05-09T16:01:22.601060Z File "/usr/local/lib/python3.11/site-packages/gevent/thread.py", line 132, in acquire
DEFAULT 2024-05-09T16:01:22.604530Z sleep()
DEFAULT 2024-05-09T16:01:22.604547Z File "/usr/local/lib/python3.11/site-packages/gevent/hub.py", line 159, in sleep
DEFAULT 2024-05-09T16:01:22.606630Z waiter.get()
DEFAULT 2024-05-09T16:01:22.606674Z File "src/gevent/_waiter.py", line 143, in gevent._gevent_c_waiter.Waiter.get
DEFAULT 2024-05-09T16:01:22.606865Z File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
DEFAULT 2024-05-09T16:01:22.606906Z File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
DEFAULT 2024-05-09T16:01:22.606963Z File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
DEFAULT 2024-05-09T16:01:22.607023Z File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
DEFAULT 2024-05-09T16:01:22.607071Z File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
DEFAULT 2024-05-09T16:01:22.607131Z greenlet.GreenletExit:

已尝试的修复动作 #

• 将Locust升级至2.17.0（曾发现类似Bug，升级后问题仍存在）
• 将Locust升级至2.27.0
• 将gevent升级至24.2.1

更新：可复现问题的最简测试代码 #

import google.oauth2.id_token
from config import config
from locust import HttpUser, task, between

BASE_URL = config.BASE_URL


def set_header():
    """Set header for SDS requests"""
    auth_req = google.auth.transport.requests.Request()
    auth_token = google.oauth2.id_token.fetch_id_token(
        auth_req, audience=config.OAUTH_CLIENT_ID
    )
    return {"Authorization": f"Bearer {auth_token}"}

HEADERS = set_header()

class PerformanceTests(HttpUser):
    wait_time = between(1, 2)
    host = config.BASE_URL

    # Test unit data endpoint
    @task
    def http_get_data_v1(self):
        self.client.get(
            f"{config.BASE_URL}/v1/unit_data?dataset_id=2cf4e4e6-ba69-4935-bf57-7b72c7bc2a32&identifier={config.FIXED_IDENTIFIERS[0]}",
            headers=HEADERS,
        )

Locust运行日志 #

[2024-05-13 12:59:23,139] localhost/INFO/locust.main: Starting web interface at http://0.0.0.0:8089
[2024-05-13 12:59:23,157] localhost/INFO/locust.main: Starting Locust 2.27.0
[2024-05-13 12:59:40,825] localhost/INFO/locust.runners: Ramping to 200 users at a rate of 100.00 per second
[2024-05-13 12:59:42,847] localhost/INFO/locust.runners: All users spawned: {"PerformanceTests": 200} (200 total users)
[2024-05-13 13:00:34,706] localhost/INFO/locust.runners: Ramping to 200 users at a rate of 100.00 per second
[2024-05-13 13:00:36,820] localhost/INFO/locust.runners: All users spawned: {"PerformanceTests": 200} (200 total users)
[2024-05-13 13:01:05,715] localhost/INFO/locust.runners: Ramping to 200 users at a rate of 100.00 per second
[2024-05-13 13:01:07,766] localhost/INFO/locust.runners: All users spawned: {"PerformanceTests": 200} (200 total users)

内容的提问来源于stack exchange，提问作者I. Lai