Flink CDC + Debezium Oracle 部分表报'db history topic or its content is fully or partially missing'错误排查求助
Flink CDC + Debezium Oracle 部分表报'db history topic or its content is fully or partially missing'错误排查求助
大家好,我现在用Flink搭配Debezium,通过LogMiner来消费Oracle数据库的CDC变更,有些表运行完全正常,比如下面这个表就没出现任何问题:
CREATE TABLE CDC_PRODUCTS ( ID NUMBER, NAME VARCHAR2(100), DESCRIPTION VARCHAR2(200), WEIGHT NUMBER(10, 3) );
但有另一部分表会抛出如下错误:
io.debezium.DebeziumException: The db history topic or its content is fully or partially missing. Please check database history topic configuration and re-execute the snapshot. Full stack trace: io.debezium.DebeziumException: The db history topic or its content is fully or partially missing. Please check database history topic configuration and re-execute the snapshot. at io.debezium.relational.HistorizedRelationalDatabaseSchema.recover(HistorizedRelationalDatabaseSchema.java:59) at org.apache.flink.cdc.connectors.oracle.source.reader.fetch.OracleSourceFetchTaskContext.validateAndLoadDatabaseHistory(OracleSourceFetchTaskContext.java:275) at org.apache.flink.cdc.connectors.oracle.source.reader.fetch.OracleSourceFetchTaskContext.configure(OracleSourceFetchTaskContext.java:118) at org.apache.flink.cdc.connectors.base.source.reader.external.IncrementalSourceStreamFetcher.submitTask(IncrementalSourceStreamFetcher.java:84) at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceSplitReader.submitStreamSplit(IncrementalSourceSplitReader.java:261) at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceSplitReader.pollSplitRecords(IncrementalSourceSplitReader.java:153) at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceSplitReader.fetch(IncrementalSourceSplitReader.java:98) at org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58) at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:162) at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:114) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) 20:13:49.779 [Source Data Fetcher for Source: OracleParallelSource (1/1)#92] ERROR org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager - Received uncaught exception. java.lang.RuntimeException: SplitFetcher thread 0 received unexpected exception while polling the records at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:165) at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:114) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.io.IOException: io.debezium.DebeziumException: The db history topic or its content is fully or partially missing. Please check database history topic configuration and re-execute the snapshot. at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceSplitReader.fetch(IncrementalSourceSplitReader.java:101) at org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58) at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:162) ... 7 more
出问题的表有这些特点:
- 包含11个字段
- 有主键约束
- 有6个NOT NULL约束
补充一下这个问题表的DDL:
CREATE TABLE "SI_HCMP_REPLICA1"."*****_HISTORY" ( "****_ID" NUMBER DEFAULT TO_NUMBER(TO_CHAR(SYSDATE@!,'YYYYMMDDHH24MI')||TO_CHAR(****.****.NEXTVAL)), "CUSTOMER_ID" NUMBER NOT NULL ENABLE, "ASSET_ID" NUMBER NOT NULL ENABLE, "ASSET_LAST_MODIFIED_ON" DATE NOT NULL ENABLE, "CREATED_ON" DATE DEFAULT SYSDATE NOT NULL ENABLE, "CREATED_BY_ID" VARCHAR2(100), "UPDATED_ON" DATE, "UPDATED_BY_ID" VARCHAR2(100), "PROCESS_STATUS" VARCHAR2(1) DEFAULT 'N' NOT NULL ENABLE, "ASSET_TECH_ID" NUMBER, "WORKFLOW_FLAG" VARCHAR2(1) DEFAULT 'N' NOT NULL ENABLE, CONSTRAINT "RM_ASSET_HISTORY_PK" PRIMARY KEY ("****_ID") USING INDEX PCTFREE
现在有几个疑问想请教大家:
- 这个问题会不会是因为特定表缺少快照或者元数据导致的?
- 当表存在PRIMARY KEY或者NOT NULL这类约束时,Debezium LogMiner的表现会有差异吗?
- 这种情况有没有什么恢复或者重新初始化DB history的建议?
非常感谢大家的帮助!
内容来源于stack exchange




