Docker部署Django+Kafka消费者时重复执行问题求助

问题现象 #

搭建集成Django和Kafka消费者的新项目，使用confluent-python实现消费者。通过docker-compose up启动时，Django管理命令代码执行两次，日志Starting consumer打印两次，同时出现两个消费者连接：

• 尝试将代码作为独立Python脚本（配置Django环境）和Django管理命令运行，问题依旧
• 即使简化代码为仅打印一条日志，仍会执行两次

相关代码与配置 #

Django管理命令代码 #

import simplejson
from confluent_kafka import Consumer
from django.core.management import BaseCommand
from loguru import logger
from apps.<my_service> import KafkaHandler


class Command(BaseCommand):
    help = "Start Reporting Kafka Consumer"

    def handle(self, *args, **kwargs):
        logger.debug("Starting consumer")
        consumer = Consumer(
            {"bootstrap.servers": "kafka:19091", "group.id": "reporting"}
        )
        consumer.subscribe(["test_producer"])
        kafka_handler = KafkaHandler()

        try:
            while True:
                logger.debug("Consuming")
                msg = consumer.poll(1.0)

                if msg is None:
                    continue
                if msg.error():
                    logger.debug(f"Consumer error: {msg.error()}")
                    continue

                event_message = msg.value().decode()
                logger.debug(f"Received message: {event_message}")
                json_message = simplejson.loads(event_message)
                kafka_handler.handle(json_message)

        finally:
            consumer.close()

Docker Entrypoint脚本 #

#!/bin/bash

set -e

if [[ "${PROJECT_RUN_MODE}" == "production" ]]; then
    exec <run production mode>
elif [[ "${PROJECT_RUN_MODE}" == "develop" ]]; then
    if [[ $# == 0 ]]; then
        echo "Starting dev server..."
        exec python "${PROJECT_DIR}/manage.py" start_consumer
    else
        exec "$@"
    fi
else
    echo "Unknown run mode (${PROJECT_RUN_MODE}). Aborting..."
    exit 1
fi

Dockerfile #

FROM python:3.11.1-slim as compile-image

# production / develop
ARG DEPLOYMENT_MODE="production"
ARG BASE_DIR="/opt/project"
ARG PROJECT_DIR="${PROJECT_DIR}/project_dir"
ARG DJANGO_SETTINGS_MODULE="settings.settings"
ARG DELETE_STATIC=0

ENV BASE_DIR ${BASE_DIR}
ENV PROJECT_DIR ${PROJECT_DIR}
ENV PROJECT_PORT ${PROJECT_PORT}
ENV PHOENIX_RUN_MODE ${DEPLOYMENT_MODE}
ENV DJANGO_SETTINGS_MODULE ${DJANGO_SETTINGS_MODULE}
ENV PYTHONUNBUFFERED 1
ENV DELETE_STATIC ${DELETE_STATIC}
ENV PATH="/opt/venv/bin:$PATH"

WORKDIR "${PROJECT_DIR}"
## Add code
COPY . "${PROJECT_DIR}"

RUN apt-get update && \
    apt-get -y install g++ libjpeg-dev zlib1g-dev libffi-dev default-mysql-client default-libmysqlclient-dev musl-dev curl && \
    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | bash -s -- -y && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* && \
    python -m venv /opt/venv && \
    pip install --upgrade pip setuptools wheel && \
    # install requirements
    pip install -r "settings/requirements/${DEPLOYMENT_MODE}.txt"


FROM python:3.11.1-slim as run-image

# production / develop

ARG DEPLOYMENT_MODE="production"
ARG BASE_DIR="/opt/project"
ARG PROJECT_DIR="${BASE_DIR}/project_dir"
ARG PHOENIX_PORT="80"
ARG DJANGO_SETTINGS_MODULE="settings.settings"
ARG STOPSIGNAL="SIGTERM"

ENV BASE_DIR ${BASE_DIR}
ENV PROJECT_DIR ${PROJECT_DIR}
ENV PROJECT_PORT ${PROJECT_PORT}
ENV PHOENIX_RUN_MODE ${DEPLOYMENT_MODE}
ENV DJANGO_SETTINGS_MODULE ${DJANGO_SETTINGS_MODULE}
ENV PYTHONUNBUFFERED 1
ENV PATH="/opt/venv/bin:$PATH"

COPY --from=compile-image /opt/venv /opt/venv
RUN apt-get update && \
    apt-get -y install supervisor g++ gettext libffi-dev default-mysql-client default-libmysqlclient-dev musl-dev && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* && \
    mkdir -p /var/log/supervisor

WORKDIR "${PROJECT_DIR}"
## Add code
COPY . "${PROJECT_DIR}"


# Collect static files directly on the image if the build arg is enabled
RUN chmod +x "${PROJECT_DIR}/docker/docker-entrypoint.sh"  && \
    cp "${PROJECT_DIR}/docker/supervisord.conf" /etc/supervisor/conf.d/supervisord.conf

# Define entry point
ENTRYPOINT ["${PROJECT_DIR}/docker/docker-entrypoint.sh"]

# Expose server port and declare signal for gracefully stopping the container
EXPOSE ${PROJECT_PORT}
STOPSIGNAL ${STOPSIGNAL}

docker-compose.yml #

version: "3"

services:
  phoenix:
    entrypoint: ["bash", "docker/docker-entrypoint.sh"]
    build:
      context: ..
      dockerfile: docker/Dockerfile
      args:
        DEPLOYMENT_MODE: develop
    stop_signal: SIGINT
    environment:
    <envs>
    ports:
    - <ports>

问题原因与解决方案 #

核心原因 #

1. Docker Entrypoint重复定义：Dockerfile中已声明ENTRYPOINT，但docker-compose.yml又再次指定entrypoint，导致entrypoint脚本被执行两次，进而触发两次管理命令调用
2. Supervisor配置冲突：Dockerfile中安装了supervisor并复制了配置文件，若supervisord.conf中也配置了启动消费者命令，会与entrypoint的执行逻辑重复

解决方案 #

方案1：移除docker-compose中的重复entrypoint配置

修改docker-compose.yml，删除entrypoint行，保留Dockerfile中定义的entrypoint即可：

version: "3"

services:
  phoenix:
    build:
      context: ..
      dockerfile: docker/Dockerfile
      args:
        DEPLOYMENT_MODE: develop
    stop_signal: SIGINT
    environment:
    <envs>
    ports:
    - <ports>

方案2：清理Supervisor冗余配置

如果不使用supervisor管理进程，可在Dockerfile中移除相关安装与配置步骤：

# 移除以下相关行
RUN apt-get update && \
    apt-get -y install supervisor ... && \
    mkdir -p /var/log/supervisor
RUN cp "${PROJECT_DIR}/docker/supervisord.conf" /etc/supervisor/conf.d/supervisord.conf

若需使用supervisor，检查supervisord.conf，确保没有重复配置start_consumer命令

方案3：验证启动逻辑与环境变量

确认PROJECT_RUN_MODE环境变量正确设置为develop，避免entrypoint脚本分支逻辑错误；同时确保entrypoint中的exec命令正确替换当前进程，避免子进程残留导致重复执行

内容的提问来源于stack exchange，提问作者Pedro Silva