Locust压测用户量增长引发MemoryError故障求助

错误栈信息 #

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\o-ddong\work\test\ - locust\venv\lib\site-packages\locust\user\task.py", line 347, in run
    self.execute_next_task()
  File "d:\o-ddong\work\test\ - locust\venv\lib\site-packages\locust\user\task.py", line 372, in execute_next_task
    self.execute_task(self._task_queue.pop(0))
  File "d:\o-ddong\work\test\ - locust\venv\lib\site-packages\locust\user\task.py", line 493, in execute_task
    task(self.user)
  File "D:\o-ddong\Work\test\ - locust\send.py", line 37, in diagnose_report
    req = self.client.post("/diagnose/", data=data.encode('utf-8'), headers=headers, name="diagnose")
  File "d:\o-ddong\work\test\ - locust\venv\lib\site-packages\locust\contrib\fasthttp.py", line 280, in post
    return self.request("POST", url, data=data, **kwargs)
  File "d:\o-ddong\work\test\ - locust\venv\lib\site-packages\locust\contrib\fasthttp.py", line 215, in request
    response = self._send_request_safe_mode(method, built_url, payload=data, headers=headers, **kwargs)
  File "d:\o-ddong\work\test\ - locust\venv\lib\site-packages\locust\contrib\fasthttp.py", line 128, in _send_request_safe_mode
    return self.client.urlopen(url, method=method, **kwargs)
  File "d:\o-ddong\work\test\ - locust\venv\lib\site-packages\geventhttpclient\useragent.py", line 459, in urlopen
    last_error = self._handle_error(e, url=req.url)
  File "d:\o-ddong\work\test\ - locust\venv\lib\site-packages\geventhttpclient\useragent.py", line 424, in _handle_error
    raise reraise(type(e), e, sys.exc_info()[2])
  File "d:\o-ddong\work\test\ - locust\venv\lib\site-packages\six.py", line 719, in reraise
    raise value
  File "d:\o-ddong\work\test\ - locust\venv\lib\site-packages\geventhttpclient\useragent.py", line 454, in urlopen
    resp = self._urlopen(req)
  File "d:\o-ddong\work\test\ - locust\venv\lib\site-packages\locust\contrib\fasthttp.py", line 546, in _urlopen
    client = self.clientpool.get_client(request.url_split)
  File "d:\o-ddong\work\test\ - locust\venv\lib\site-packages\geventhttpclient\client.py", line 306, in get_client
    client = HTTPClient.from_url(url, **self.client_args)
  File "d:\o-ddong\work\test\ - locust\venv\lib\site-packages\geventhttpclient\client.py", line 68, in from_url
    return cls(url.host, port=url.port, ssl=enable_ssl, **kw)
  File "d:\o-ddong\work\test\ - locust\venv\lib\site-packages\geventhttpclient\client.py", line 107, in __init__
    self._connection_pool = SSLConnectionPool(
  File "d:\o-ddong\work\test\ - locust\venv\lib\site-packages\geventhttpclient\connectionpool.py", line 241, in __init__
    self.init_ssl_context(ssl_context_factory)
  File "d:\o-ddong\work\test\ - locust\venv\lib\site-packages\geventhttpclient\connectionpool.py", line 256, in init_ssl_context
    self.ssl_context = ssl_context_factory()
  File "d:\o-ddong\work\test\ - locust\venv\lib\site-packages\locust\contrib\fasthttp.py", line 65, in insecure_ssl_context_factory
    context = gevent.ssl.create_default_context()
  File "C:\Users\test\AppData\Local\Programs\Python\Python38-32\lib\ssl.py", line 750, in create_default_context
    context.load_default_certs(purpose)
  File "C:\Users\test\AppData\Local\Programs\Python\Python38-32\lib\ssl.py", line 574, in load_default_certs
    self._load_windows_store_certs(storename, purpose)
  File "C:\Users\test\AppData\Local\Programs\Python\Python38-32\lib\ssl.py", line 562, in _load_windows_store_certs
    certs.extend(cert)
MemoryError

故障描述 #

运行Locust压测时，随着并发用户量增加，出现内存耗尽问题，导致压测直接终止。

预期行为 #

Locust可稳定持续运行，不会因用户量增长至设定上限而停止。

实际行为 #

触发上述错误栈中的MemoryError异常，压测进程终止。

复现步骤 #

使用以下简化代码执行压测：

from locust import FastHttpUser, task, constant


class User(FastHttpUser):
    host = "hostURL"

    @task(1)
    def send(self):
        headers = {}
        req = self.client.get("URL", headers=headers)
    
    wait_time = constant(1)

环境信息 #

• 本地PC：i5-9600K，32GB内存
• Python版本：3.8（32位）
• Locust版本：2.14.2
• 测试基础设施：10台m5.large服务器

排查思路 #

1. 32位Python内存限制：32位Python进程最大只能使用约2GB内存，即便机器有32GB内存也无法突破这个限制，这是触发MemoryError的核心原因之一。
2. SSL上下文重复加载：FastHttpUser在创建新连接池时，会重复调用create_default_context()加载系统证书存储中的所有证书，随着并发连接数增加，大量重复加载的证书占用内存飙升。
3. 连接池配置不合理：默认连接池参数可能允许创建过多并发连接，每个连接都需要占用内存资源，累积后导致内存耗尽。

解决方案 #

1. 切换到64位Python
   直接替换32位Python为对应版本的64位安装包，64位Python可以充分利用机器的32GB内存，从根本上解决进程内存上限问题。

2. 复用SSL上下文
   自定义SSL上下文工厂，提前加载一次系统证书并复用，避免每次创建连接池都重复加载。示例代码：

   import gevent.ssl
   # 提前创建并复用SSL上下文
   _shared_ssl_context = None
   
   def shared_ssl_context_factory():
       global _shared_ssl_context
       if _shared_ssl_context is None:
           _shared_ssl_context = gevent.ssl.create_default_context()
           # 如果是忽略证书验证的场景，可以添加：
           # _shared_ssl_context.check_hostname = False
           # _shared_ssl_context.verify_mode = gevent.ssl.CERT_NONE
       return _shared_ssl_context
   
   class User(FastHttpUser):
       host = "hostURL"
       # 指定复用的SSL上下文工厂
       ssl_context_factory = shared_ssl_context_factory
   
       @task(1)
       def send(self):
           headers = {}
           req = self.client.get("URL", headers=headers)
       
       wait_time = constant(1)
   

3. 优化连接池参数
   在FastHttpUser中配置连接池的最大连接数，减少不必要的并发连接占用内存：

   class User(FastHttpUser):
       host = "hostURL"
       # 设置单连接池最大连接数
       max_connections = 100
       # 设置每个用户的最大并发请求数
       max_concurrent_requests = 10
   
       @task(1)
       def send(self):
           headers = {}
           req = self.client.get("URL", headers=headers)
       
       wait_time = constant(1)
   

4. 临时切换到HttpUser
   如果FastHttpUser的问题暂时无法解决，可以改用基于requests库的HttpUser，虽然性能略低于FastHttpUser，但内存管理更稳定，适合临时验证压测逻辑：

   from locust import HttpUser, task, constant
   
   class User(HttpUser):
       host = "hostURL"
   
       @task(1)
       def send(self):
           headers = {}
           req = self.client.get("URL", headers=headers)
       
       wait_time = constant(1)
   

内容的提问来源于stack exchange，提问作者ddangdang