节点异常重启时Kafka-Connect插件配置丢失问题排查求助

First, let's start with your Dockerfile for reference:

FROM confluentinc/cp-kafka-connect:5.3.1
ENV CONNECT_PLUGIN_PATH=/usr/share/java
# JDBC-MariaDB
RUN wget -nv -P /usr/share/java/kafka-connect-jdbc/ https://downloads.mariadb.com/Connectors/java/connector-java-2.4.4/mariadb-java-client-2.4.4.jar
# SNMP Source
RUN wget -nv -P /tmp/ https://github.com/name/kafka-connect-snmp/releases/download/0.0.1.11/kafka-connect-snmp-0.0.1.11.tar.gz
RUN mkdir /tmp/kafka-connect-snmp && tar -xf /tmp/kafka-connect-snmp-0.0.1.11.tar.gz -C /tmp/kafka-connect-snmp/
RUN mv /tmp/kafka-connect-snmp/usr/share/kafka-connect/kafka-connect-snmp /usr/share/java/

Based on this setup, here are the most likely causes and actionable checks to fix the problem:

Possible Root Causes #

• Volume Mount Overriding Plugin Directory: If you're mounting a host volume or persistent volume to /usr/share/java (the path defined in CONNECT_PLUGIN_PATH), this will replace the directory that contains your plugins from the image. After a restart, the mounted volume might be empty or not retain the plugin files.
• Incorrect Deployment Image: If your orchestration tool (Kubernetes, Docker Swarm, etc.) is pulling the base confluentinc/cp-kafka-connect:5.3.1 image instead of your custom built image after a restart, the plugins won't be present.
• Permission Misconfiguration: The Kafka Connect process runs as a non-root user by default in Confluent images. If the plugin files/directories have incorrect permissions, the process can't read them after a restart.
• File System Corruption: If using bind mounts from the node's local storage, an abnormal shutdown might have corrupted the plugin files, making them undetectable to Kafka Connect.

Step-by-Step Troubleshooting & Verification #

1. Confirm Your Custom Image Includes the Plugins #

First, make sure your built image actually has the plugins baked in:

• Spin up a temporary container from your custom image:

  docker run --rm -it YOUR_CUSTOM_IMAGE_TAG bash
  

• Check the plugin directories:

  ls -l /usr/share/java/kafka-connect-jdbc/
  ls -l /usr/share/java/kafka-connect-snmp/
  

You should see mariadb-java-client-2.4.4.jar and the SNMP plugin files here. If not, your image build failed—re-run the build and check the output for errors (build logs should be accessible even without runtime logs).

2. Check for Volume Mounts Overriding the Plugin Path #

Volume mounts can hide the plugin files from your image. Verify if any volume is mounted to /usr/share/java:

• For Docker, inspect the container:

  docker inspect YOUR_CONNECT_CONTAINER | grep -A 10 "Mounts"
  

• For Kubernetes, check your Deployment/StatefulSet YAML for volume mounts targeting /usr/share/java.

If you find such a mount, you have two fixes:

• Remove the volume mount (since plugins are baked into the image, persistent storage here isn't necessary unless you need to add plugins dynamically)
• Use an init container or startup script to copy plugins from the image to the mounted volume on each restart.

3. Verify Deployment is Using the Correct Image #

Ensure your orchestration isn't falling back to the base Confluent image:

• For Docker, check docker ps or your docker-compose.yml to confirm the image name matches your custom build.
• For Kubernetes, look at the image field in your Pod template—make sure it's pointing to your tagged custom image, not confluentinc/cp-kafka-connect:5.3.1.

If it's using the base image, update the configuration to use your custom build.

4. Check File Permissions in the Running Container #

If the container uses your custom image but plugins are missing, permissions might be the issue:

• Exec into the running container:

  docker exec -it YOUR_CONNECT_CONTAINER bash
  

• Check ownership and access rights:

  ls -ld /usr/share/java /usr/share/java/kafka-connect-jdbc /usr/share/java/kafka-connect-snmp
  ls -l /usr/share/java/kafka-connect-jdbc/mariadb-java-client-2.4.4.jar
  

The default user for Confluent's Kafka Connect image is appuser. If this user doesn't have read access, add this line to your Dockerfile to fix permissions during build:

RUN chown -R appuser:appuser /usr/share/java/kafka-connect-jdbc /usr/share/java/kafka-connect-snmp

5. Confirm CONNECT_PLUGIN_PATH Isn't Overridden #

Double-check that the plugin path environment variable is set correctly at runtime:

• Run this command in the running container:

  docker exec YOUR_CONNECT_CONTAINER printenv CONNECT_PLUGIN_PATH
  

It should return /usr/share/java. If it's set to a different path, Kafka Connect is looking for plugins elsewhere—check your deployment config for any runtime environment variable overrides.

6. Inspect Bind Mounts for Corruption (If Used) #

If you're using a host bind mount for /usr/share/java, check the host directory on the node:

ls -l /PATH/TO/YOUR/HOST/BIND/MOUNT

If files are missing or corrupted, re-copy the plugins into the host directory, or switch to using the plugins baked into your image instead of the bind mount.

内容的提问来源于stack exchange，提问作者SomeGuyWhoCodes