Flink CDC同步Postgres至StarRocks时Postgres JAR包缺失问题
搭建Flink CDC管道时无法识别Postgres CDC连接器的问题
我正在基于Flink搭建CDC管道,将PostgreSQL数据库的数据同步至StarRocks数据库。目前JobManager和TaskManager已正常启动,但启动CDC任务时始终报错:
Exception in thread "main" java.lang.RuntimeException: Cannot find factory with identifier "postgres-cdc" in the classpath. Available factory classes are: org.apache.flink.cdc.connectors.starrocks.sink.StarRocksDataSinkFactory ...
运行时能找到StarRocks连接器的JAR包,但同一目录(/opt/flink/usrlib/flink-cdc-3.4.0/lib/)下的PostgreSQL相关JAR包无法被识别。我是首次使用Flink,已经花了一天时间尝试配置但没解决,求解决方案。
容器配置
# JobManager podman run \ --name=flink_jobmanager \ --detach \ --env FLINK_PROPERTIES="jobmanager.rpc.address: 192.168.1.10" \ --volume "/var/opt/flink/data:/opt/flink/usrlib" \ --network flink_net \ --publish 8081:8081 \ --publish 6123:6123 \ docker.io/flink:latest \ jobmanager # TaskManager podman run \ --name=flink_taskmanager \ --detach \ --env FLINK_PROPERTIES="jobmanager.rpc.address: 192.168.1.10" \ --volume "/var/opt/flink/data:/opt/flink/usrlib" \ --network flink_net \ docker.io/flink:latest \ taskmanager # Start the job podman run \ --rm \ --env FLINK_PROPERTIES="jobmanager.rpc.address: 192.168.1.10" \ --volume "/var/opt/flink/data:/opt/flink/usrlib:rw" \ --network flink_net \ docker.io/flink:latest \ /opt/flink/usrlib/flink-cdc-3.4.0/bin/flink-cdc.sh \ /opt/flink/usrlib/postgres-to-starrocks.yaml
目录结构与JAR包
PostgreSQL JAR包参考官方文档下载,本地目录结构如下:
/var/opt/flink/data$ tree . ├── flink-cdc-3.4.0 │ ├── bin │ │ └── flink-cdc.sh │ ├── conf │ │ ├── flink-cdc.yaml │ │ └── log4j-cli.properties │ ├── lib │ │ ├── flink-cdc-dist-3.4.0.jar │ │ ├── flink-cdc-pipeline-connector-starrocks-3.4.0.jar │ │ ├── flink-connector-postgres-cdc-3.4.0.jar │ │ └── flink-sql-connector-postgres-cdc-3.4.0.jar │ ├── LICENSE │ ├── log │ └── NOTICE └── postgres-to-starrocks.yaml
同步任务配置文件(postgres-to-starrocks.yaml)
# Job properties job: name: postgres-to-starrocks parallelism: 4 checkpointing: enabled: true interval: 60000 # Checkpoint every 60 seconds (in milliseconds) mode: EXACTLY_ONCE timeout: 600000 # Checkpoint timeout 10 minutes min-pause: 5000 # Minimum pause between checkpoints source: type: "postgres-cdc" hostname: 192.168.1.10 port: 5432 username: username password: "password" database-name: database schema-name: public tables: database.\.* sink: type: starrocks name: StarRocks Sink jdbc-url: "jdbc:mysql://192.168.1.10:9030" load-url: 192.168.1.10:8030 username: username password: "password" database-name: test
内容的提问来源于stack exchange,提问作者usdn
相关产品推荐
相关产品推荐

