Debezium结合PostgreSQL与Redis Streams运行异常求助

在部署Debezium搭配Redis和PostgreSQL时出现启动错误，Redis Insights与PostgreSQL可正常访问，但Debezium无法启动。相关配置及错误信息如下：

Docker Compose 配置 #

version: "3.3"
services:
  redis-stack:
    image: redis/redis-stack:7.0.6-RC4
    restart: unless-stopped
    ports:
      - 10001:6379
      - 13333:8001
    volumes:
      - ./data/redis-stack/:/data

  db:
    image: postgres
    restart: unless-stopped
    ports:
      - 5432:5432
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: 1234

  pgadmin:
    image: dpage/pgadmin4
    restart: unless-stopped
    ports:
      - 5555:80
    environment:
      PGADMIN_DEFAULT_PASSWORD: 1234
      PGADMIN_DEFAULT_EMAIL: arkan.m.gerges@gmail.com

  debezium:
    image: debezium/server:2.1.2.Final
    restart: unless-stopped
    ports:
      - 8180:8080
    volumes:
      - ./config/debezium:/debezium/conf
      - ./data/debezium:/debezium/data
    depends_on:
      - redis-stack
      - db

networks:
    app-network:

Debezium 配置文件（config/debezium/application.properties） #

debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector
debezium.source.offset.storage=io.debezium.storage.redis.offset.RedisOffsetBackingStore
debezium.source.offset.flush.interval.ms=0
debezium.source.offset.storage.redis.address=redis-stack:6379
debezium.source.schema.history.internal=io.debezium.storage.redis.history.RedisSchemaHistory
debezium.source.schema.history.internal.redis.address=redis-stack:6379
debezium.sink.type=redis
debezium.sink.redis.address=redis-stack:6379
debezium.source.database.hostname=db
debezium.source.database.port=5432
debezium.source.database.user=postgres
debezium.source.database.password=1234
debezium.source.database.dbname=softwaredev_expert
debezium.source.database.server.name=db
debezium.source.schema.whitelist=public
debezium.source.schema.include.list=public
debezium.source.plugin.name=pgoutput

错误日志（翻译后） #

2023-10-12 10:00:00,000 ERROR [io.debezium.server.Main] (main) 启动Debezium Server失败: java.lang.IllegalArgumentException: 无法找到或加载Redis sink连接器类 'io.debezium.sink.redis.RedisSinkConnector'
    at io.debezium.server.sink.SinkConnectorFactory.create(SinkConnectorFactory.java:62)
    at io.debezium.server.DebeziumServer.start(DebeziumServer.java:123)
    at io.debezium.server.Main.main(Main.java:77)
引起原因: java.lang.ClassNotFoundException: io.debezium.sink.redis.RedisSinkConnector
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:467)
    at io.debezium.server.sink.SinkConnectorFactory.create(SinkConnectorFactory.java:59)
    ... 2 more

问题分析与解决步骤 #

1. 缺失Redis Sink依赖包 #

Debezium Server官方镜像默认不包含Redis Sink连接器的依赖，导致启动时找不到对应类。解决方式二选一：

方式一：自定义Debezium镜像

创建Dockerfile：

FROM debezium/server:2.1.2.Final
RUN mkdir -p /debezium/lib
RUN curl -L https://repo1.maven.org/maven2/io/debezium/debezium-connector-redis/2.1.2.Final/debezium-connector-redis-2.1.2.Final.jar -o /debezium/lib/debezium-connector-redis-2.1.2.Final.jar
RUN curl -L https://repo1.maven.org/maven2/io/lettuce/lettuce-core/6.2.6.RELEASE/lettuce-core-6.2.6.RELEASE.jar -o /debezium/lib/lettuce-core-6.2.6.RELEASE.jar

修改docker-compose中debezium服务的配置，替换为构建自定义镜像：

debezium:
  build: .
  restart: unless-stopped
  ports:
    - 8180:8080
  volumes:
    - ./config/debezium:/debezium/conf
    - ./data/debezium:/debezium/data
  depends_on:
    - redis-stack
    - db

方式二：挂载本地依赖包

本地创建./lib目录，下载以下两个jar包放入该目录：

• debezium-connector-redis-2.1.2.Final.jar
• lettuce-core-6.2.6.RELEASE.jar

修改docker-compose中debezium服务的volumes，添加lib目录挂载：

volumes:
  - ./config/debezium:/debezium/conf
  - ./data/debezium:/debezium/data
  - ./lib:/debezium/lib

2. 配置PostgreSQL逻辑复制 #

pgoutput插件需要PostgreSQL开启逻辑复制，修改docker-compose的db服务配置：

db:
  image: postgres
  restart: unless-stopped
  ports:
    - 5432:5432
  environment:
    POSTGRES_USER: postgres
    POSTGRES_PASSWORD: 1234
  command: -c wal_level=logical -c max_wal_senders=10 -c max_replication_slots=10
  volumes:
    - ./data/postgres:/var/lib/postgresql/data

进入PostgreSQL执行以下SQL，确保目标数据库存在且用户有复制权限：

CREATE DATABASE softwaredev_expert;
ALTER USER postgres WITH REPLICATION;

3. 清理冗余配置 #

配置文件中debezium.source.schema.whitelist=public与debezium.source.schema.include.list=public重复，保留其中一个即可（新版本推荐使用schema.include.list）。

内容的提问来源于stack exchange，提问作者Arkan