Celery+Django+Selenium应用出现大量僵尸进程，如何清理任务资源并解决该问题？

我正在构建一个Celery + Django + Selenium的应用，用Celery来运行基于Selenium的浏览器进程。使用的版本如下：

celery==5.2.6
redis==3.4.1
selenium-wire==5.1.0
Django==4.0.4
djangorestframework==3.13.1

运行几个小时后，我发现应用会生成数千个僵尸进程，而且确定问题与Celery Docker容器相关——因为执行sudo /usr/local/bin/docker-compose -f /data/new_app/docker-compose.yml restart celery命令重启Celery容器后，所有僵尸进程就都消失了。

我的代码 #

Web服务端点代码

from rest_framework.decorators import api_view

@api_view(['POST'])
def periodic_check_all_urls(request): # web-service endpoint
    ...
    check_urls.delay(parsing_results_ids) # call celery task

Celery任务代码

from celery import shared_task


@shared_task()
def check_urls(parsing_result_ids: List[int]):
    """
    Run Selenium-based parser
    the parser exctracts data and saves in database
    """
    try:
        logger.info(f"{datetime.now()} Start check_urls")
        parser = Parser() # open selenium browser
        parsing_results = ParsingResult.objects.filter(pk__in=parsing_result_ids).exclude(status__in=["DONE", "FAILED"])
        parser.check_parsing_result(parsing_results)
    except Exception as e:
        full_trace = traceback.format_exc()
    finally:
        if 'parser' in locals():
            parser.stop()

Selenium浏览器的停止函数与析构方法

class Parser():
    def __init__(self):
        """
        Prepare parser
        """
        if not USE_GUI:
            self.display = Display(visible=0, size=(800, 600))
            self.display.start()
        

        """ Replaced with FireFox
        self.driver = get_chromedriver(proxy_data)
        """
        proxy_data = {
            ...
        }
        self.driver = get_firefox_driver(proxy_data=proxy_data)

    
    def __del__(self):
        self.stop()

    def stop(self):
        try:
            self.driver.quit()
            logger.info("Selenium driver closed")
        except:
            pass
        
        try:
            self.display.stop()
            logger.info("Display stopped")
        except:
            pass

尝试过的优化（未解决僵尸进程问题） #

我试过调整Celery的任务资源限制和超时设置，但没有效果：

Django settings.py中的Celery配置

# celery setting (documents generation)
CELERY_BROKER_URL = os.environ.get("CELERY_BROKER", "redis://redis:6379/0")
CELERY_RESULT_BACKEND = os.environ.get("CELERY_BROKER", "redis://redis:6379/0")
CELERY_IMPORTS = ("core_app.celery",)
CELERY_TASK_TIME_LIMIT = 10 * 60

Docker中的Celery配置

celery:
    build: ./project
    command: celery -A core_app worker  --loglevel=info --concurrency=15 --max-memory-per-child=1000000
    volumes:
      - ./project:/usr/src/app
      - ./project/media:/project/media
      - ./project/logs:/project/logs
    env_file:
      - .env
    environment:
    # environment variables declared in the environment section override env_file
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
      - CELERY_BROKER=redis://redis:6379/0
      - CELERY_BACKEND=redis://redis:6379/0
    depends_on:
      - django
      - redis

参考过的解决方案（未奏效） #

• 我曾查找过关于如何终止Celery任务的相关问题，但没有帮助
• 也看过Celery撤销任务后遗留僵尸进程的相关案例，不过我的任务已经包含了try/finally逻辑来清理资源

僵尸进程示例 #

执行ps aux | grep 'Z'后得到的结果：

ps aux | grep 'Z'

root     32448  0.0  0.0      0     0 ?        Z    13:45   0:00 [Utility Process] <defunct>
root     32449  0.0  0.0      0     0 ?        Z    13:09   0:00 [Utility Process] <defunct>
root     32450  0.0  0.0      0     0 ?        Z    11:13   0:00 [sh] <defunct>
root     32451  0.0  0.0      0     0 ?        Z    13:44   0:00 [Utility Process] <defunct>
root     32452  0.0  0.0      0     0 ?        Z    10:12   0:00 [Utility Process] <defunct>
root     32453  0.0  0.0      0     0 ?        Z    09:52   0:00 [sh] <defunct>
root     32454  0.0  0.0      0     0 ?        Z    10:40   0:00 [Utility Process] <defunct>
root     32455  0.0  0.0      0     0 ?        Z    09:52   0:00 [Utility Process] <defunct>
root     32456  0.0  0.0      0     0 ?        Z    10:13   0:00 [sh] <defunct>
root     32457  0.0  0.0      0     0 ?        Z    10:51   0:00 [Utility Process] <defunct>
root     32459  0.0  0.0      0     0 ?        Z    14:01   0:00 [Utility Process] <defunct>
root     32460  0.0  0.0      0     0 ?        Z    13:16   0:00 [Utility Process] <defunct>
root     32461  0.0  0.0      0     0 ?        Z    10:40   0:00 [Utility Process] <defunct>
root     32462  0.0  0.0      0     0 ?        Z    10:12   0:00 [Utility Process] <defunct>

备注：内容来源于stack exchange，提问作者mascai