App Engine迁移至Python3.11后持续出现线程错误求助

问题背景 #

我们的App Engine标准环境服务已从Python2.7迁移至Python3.11，迁移后持续出现线程相关错误，且错误信息不足以支撑调试。经排查该错误似乎不在所用Django框架范围内。

错误信息 #

错误1：Worker退出关联线程异常 #

Traceback (most recent call last):
File "/layers/google.python.runtime/python/lib/python3.11/threading.py", line 995, in _bootstrap
[2023-08-16 07:12:23 +0000] [18] [INFO] Worker exiting (pid: 18)

错误2：Logging Worker线程异常 #

Traceback (most recent call last):
  File "/layers/google.python.runtime/python/lib/python3.11/threading.py", line 995, in _bootstrap
Exception ignored in thread started byException ignored in thread started byException ignored in thread started by: : : Waiting up to 5 seconds.<bound method Thread._bootstrap of <Thread(google.cloud.logging.Worker, started daemon 68606081631808)>><bound method Thread._bootstrap of <Thread(google.cloud.logging.Worker, started daemon 68606081631808)>>

错误3：线程删除时KeyError #

Traceback (most recent call last):
  File "/layers/google.python.runtime/python/lib/python3.11/threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
  File "/layers/google.python.runtime/python/lib/python3.11/threading.py", line 1042, in _bootstrap_inner
    self._delete()
  File "/layers/google.python.runtime/python/lib/python3.11/threading.py", line 1074, in _delete
    del _active[get_ident()]

相关配置 #

app.yaml配置 #

runtime: python311
app_engine_apis: true
entrypoint: gunicorn dummy.wsgi:application -b :$PORT -w 8 -t 600

instance_class: F4

handlers:

- url: /internal/.*
  script: auto
  secure: always
  login: optional

- url: /static
  static_dir: static/
  secure: always

- url: /.*
  script: auto
  secure: always

wsgi.py配置 #

from dummy.boot import fix_path
fix_path()

import os
from django.core.wsgi import get_wsgi_application
from djangae.environment import is_production_environment
from google.appengine.api import wrap_wsgi_app

settings = "dummy.settings.live" if is_production_environment() else "dummy.settings.local"
os.environ.setdefault("DJANGO_SETTINGS_MODULE", settings)

application = get_wsgi_application()
application = wrap_wsgi_app(application)

排查思路与解决方案 #

• 调整gunicorn worker类型：
  错误关联gunicorn Worker退出日志，默认的sync worker可能与App Engine线程环境冲突。尝试切换为异步worker类型，修改entrypoint：

  entrypoint: gunicorn dummy.wsgi:application -b :$PORT -w 8 -k gevent -t 600
  

  同时在requirements.txt中添加gevent依赖。

• 修复google-cloud-logging线程兼容性：
  错误明确指向google.cloud.logging.Worker线程，可能是日志客户端版本与Python3.11不兼容。升级google-cloud-logging到最新稳定版，或改为同步日志传输：
  在Django settings的LOGGING配置中修改：

  'handlers': {
      'google_cloud': {
          'class': 'google.cloud.logging.handlers.CloudLoggingHandler',
          'transport': 'google.cloud.logging.handlers.SyncTransport',
      },
  }
  

• 检查djangae库兼容性：
  djangae主要适配App Engine Python2环境，迁移到Python3.11后可能存在线程管理冲突。尝试替换为官方App Engine Django支持方案，或暂时移除wrap_wsgi_app包装测试。

• 减少worker数量缓解资源竞争：
  当前F4实例搭配8个worker可能导致线程资源过载，尝试将worker数调整为4：

  entrypoint: gunicorn dummy.wsgi:application -b :$PORT -w 4 -t 600
  

• 开启详细调试日志：
  在app.yaml中添加环境变量增强日志输出：

  env_variables:
    PYTHONUNBUFFERED: "1"
    LOG_LEVEL: "DEBUG"
  

  同时在Django settings中配置线程相关日志的DEBUG级别，捕捉更完整的错误上下文。

内容的提问来源于stack exchange，提问作者YSC