Kafka Producer Docker应用在Windows/Linux运行失败，MAC正常求助

开发的Kafka Producer Docker应用，Docker构建环节在Windows、Linux、Mac系统均能成功完成，但只有Mac上能正常运行，Windows和Linux机器运行时抛出GSSAPI相关错误。

错误信息 #

[2022-09-06 17:53:42,174] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
 File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2051, in wsgi_app
   response = self.full_dispatch_request()
 File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1501, in full_dispatch_request
   rv = self.handle_user_exception(e)
 File "/usr/local/lib/python3.8/site-packages/flask_cors/extension.py", line 161, in wrapped_function
   return cors_after_request(app.make_response(f(*args, **kwargs)))
 File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1499, in full_dispatch_request
   rv = self.dispatch_request()
 File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1485, in dispatch_request  
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
 File "/function/model/api.py", line 21, in home
   return modelcode()
 File "/function/model/api.py", line 49, in modelcode
   producer = SerializingProducer(producer_conf)
 File "/usr/local/lib/python3.8/site-packages/confluent_kafka/serializing_producer.py", line 109, in __init__
   super(SerializingProducer, self).__init__(conf_copy)
cimpl.KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Failed to create producer: No
provider for SASL mechanism GSSAPI: recompile librdkafka with libsasl2 or openssl support. Current build options: PLAIN SASL_SCRAM OAUTHBEARER"}

Dockerfile内容 #

ARG FUNCTION_DIR="/function"
FROM python:3.8
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update &&       apt-get -y install  &&    apt-get -y install
RUN  apt-get -y install librdkafka-dev
RUN pip install --no-binary :all: confluent-kafka==0.11.0
ARG FUNCTION_DIR
RUN mkdir -p ${FUNCTION_DIR}
RUN apt-get -y install libsasl2-modules-gssapi-mit
COPY cacert.pem ${FUNCTION_DIR}
COPY CP_KAFKA_EAC_CC_DEV.keytab ${FUNCTION_DIR}
COPY ./requirements.txt ${FUNCTION_DIR}/requirements.txt
RUN  pip install -r ${FUNCTION_DIR}/requirements.txt
COPY . ${FUNCTION_DIR}/
WORKDIR ${FUNCTION_DIR}
EXPOSE 8080
ENTRYPOINT exec gunicorn --timeout 80 --bind 0.0.0.0:8080 wsgi

requirements.txt内容 #

Flask_Cors==3.0.0
python_jose==3.0.1
flask_restplus==0.11.0
Flask_JWT_Extended==3.18.0
Flask==2.0
requests==2.20.0
PyJWT==1.7.1
cryptography==35.0
py-healthcheck==1.9.0
gunicorn==20.0.4
Werkzeug~=2.0.0
jinja2==3.0.3
confluent_kafka==1.7.0
pandas==1.4.2
numpy==1.23.2
pyOpenSSL==22.0.0
kafka-python

运行命令及输出 #

$ winpty docker run -it -p 8088:8080 sample-flask-app:0.0.1
[2022-09-09 02:16:25 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2022-09-09 02:16:25 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2022-09-09 02:16:25 +0000] [1] [INFO] Using worker: sync
[2022-09-09 02:16:25 +0000] [8] [INFO] Booting worker with pid: 8
sample output
[2022-09-09 02:16:44,672] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2051, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1501, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1499, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1485, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/function/model/api.py", line 21, in home
    return modelcode()
  File "/function/model/api.py", line 49, in modelcode
    producer = SerializingProducer(producer_conf)
  File "/usr/local/lib/python3.8/site-packages/confluent_kafka/serializing_producer.py", line 109, in __init__
    super(SerializingProducer, self).__init__(conf_copy)
cimpl.KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Failed to create producer: No provider for SASL mechanism GSSAPI: recompile librdkafka with libsasl2 or openssl support. Current build options: PLAIN SASL_SCRAM OAUTHBEARER"}
^C[2022-09-09 02:16:50 +0000] [1] [INFO] Handling signal: int
[2022-09-09 02:16:50 +0000] [8] [INFO] Worker exiting (pid: 8)
[2022-09-09 02:16:50 +0000] [1] [INFO] Shutting down: Master

解决方案 #

错误核心是librdkafka编译时未包含GSSAPI支持，当前仅支持PLAIN、SASL_SCRAM、OAUTHBEARER。结合你的配置，按以下步骤修复：

1. 清理无效命令
   删除Dockerfile中两行空的apt-get -y install命令，避免冗余操作。

2. 安装完整SASL依赖
   仅安装libsasl2-modules-gssapi-mit不足以让librdkafka编译时识别GSSAPI，需先安装基础开发库。在安装librdkafka-dev前，执行：

   apt-get -y install libsasl2-dev libsasl2-modules-gssapi-mit
   

3. 统一confluent-kafka版本
   Dockerfile中手动安装的confluent-kafka==0.11.0与requirements.txt中的1.7.0版本冲突，导致依赖混乱。删除Dockerfile中的RUN pip install --no-binary :all: confluent-kafka==0.11.0，并确保安装时从源码编译：
   修改pip安装命令为：

   RUN pip install --no-binary :all: -r ${FUNCTION_DIR}/requirements.txt
   

4. 优化Dockerfile结构
   整合依赖安装步骤，减少镜像层数，确保编译librdkafka时依赖已就绪。修改后的Dockerfile示例：

   ARG FUNCTION_DIR="/function"
   FROM python:3.8
   ENV DEBIAN_FRONTEND=noninteractive
   
   # 安装系统依赖
   RUN apt-get update && \
       apt-get -y install --no-install-recommends \
           librdkafka-dev \
           libsasl2-dev \
           libsasl2-modules-gssapi-mit && \
       apt-get clean && \
       rm -rf /var/lib/apt/lists/*
   
   ARG FUNCTION_DIR
   RUN mkdir -p ${FUNCTION_DIR}
   
   # 安装Python依赖
   COPY ./requirements.txt ${FUNCTION_DIR}/requirements.txt
   RUN pip install --no-binary :all: -r ${FUNCTION_DIR}/requirements.txt
   
   # 复制应用文件
   COPY cacert.pem ${FUNCTION_DIR}
   COPY CP_KAFKA_EAC_CC_DEV.keytab ${FUNCTION_DIR}
   COPY . ${FUNCTION_DIR}/
   
   WORKDIR ${FUNCTION_DIR}
   EXPOSE 8080
   ENTRYPOINT exec gunicorn --timeout 80 --bind 0.0.0.0:8080 wsgi
   

5. 验证支持情况
   构建镜像后，进入容器执行以下命令，确认GSSAPI已被包含：

   docker exec -it <容器ID> rdkafka_conf_dump | grep sasl.mechanisms
   

内容的提问来源于stack exchange，提问作者bomsabado