You need to enable JavaScript to run this app.
最新活动
大模型
产品
解决方案
定价
生态与合作
支持与服务
开发者
了解我们

Flink CDC + Debezium Oracle 部分表报'db history topic or its content is fully or partially missing'错误排查求助

大家好,我现在用Flink搭配Debezium,通过LogMiner来消费Oracle数据库的CDC变更,有些表运行完全正常,比如下面这个表就没出现任何问题:

CREATE TABLE CDC_PRODUCTS ( 
  ID NUMBER, 
  NAME VARCHAR2(100), 
  DESCRIPTION VARCHAR2(200), 
  WEIGHT NUMBER(10, 3) 
);

但有另一部分表会抛出如下错误:

io.debezium.DebeziumException: The db history topic or its content is fully or partially missing. Please check database history topic configuration and re-execute the snapshot. 
Full stack trace: 
io.debezium.DebeziumException: The db history topic or its content is fully or partially missing. Please check database history topic configuration and re-execute the snapshot. 
	at io.debezium.relational.HistorizedRelationalDatabaseSchema.recover(HistorizedRelationalDatabaseSchema.java:59) 
	at org.apache.flink.cdc.connectors.oracle.source.reader.fetch.OracleSourceFetchTaskContext.validateAndLoadDatabaseHistory(OracleSourceFetchTaskContext.java:275) 
	at org.apache.flink.cdc.connectors.oracle.source.reader.fetch.OracleSourceFetchTaskContext.configure(OracleSourceFetchTaskContext.java:118) 
	at org.apache.flink.cdc.connectors.base.source.reader.external.IncrementalSourceStreamFetcher.submitTask(IncrementalSourceStreamFetcher.java:84) 
	at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceSplitReader.submitStreamSplit(IncrementalSourceSplitReader.java:261) 
	at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceSplitReader.pollSplitRecords(IncrementalSourceSplitReader.java:153) 
	at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceSplitReader.fetch(IncrementalSourceSplitReader.java:98) 
	at org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58) 
	at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:162) 
	at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:114) 
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) 
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) 
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 
	at java.base/java.lang.Thread.run(Thread.java:840) 

20:13:49.779 [Source Data Fetcher for Source: OracleParallelSource (1/1)#92] ERROR org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager - Received uncaught exception. 
java.lang.RuntimeException: SplitFetcher thread 0 received unexpected exception while polling the records 
	at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:165) 
	at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:114) 
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) 
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) 
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 
	at java.base/java.lang.Thread.run(Thread.java:840) 
Caused by: java.io.IOException: io.debezium.DebeziumException: The db history topic or its content is fully or partially missing. Please check database history topic configuration and re-execute the snapshot. 
	at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceSplitReader.fetch(IncrementalSourceSplitReader.java:101) 
	at org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58) 
	at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:162) 
	... 7 more

出问题的表有这些特点:

  • 包含11个字段
  • 有主键约束
  • 有6个NOT NULL约束

补充一下这个问题表的DDL:

CREATE TABLE "SI_HCMP_REPLICA1"."*****_HISTORY" ( 
  "****_ID" NUMBER DEFAULT TO_NUMBER(TO_CHAR(SYSDATE@!,'YYYYMMDDHH24MI')||TO_CHAR(****.****.NEXTVAL)), 
  "CUSTOMER_ID" NUMBER NOT NULL ENABLE, 
  "ASSET_ID" NUMBER NOT NULL ENABLE, 
  "ASSET_LAST_MODIFIED_ON" DATE NOT NULL ENABLE, 
  "CREATED_ON" DATE DEFAULT SYSDATE NOT NULL ENABLE, 
  "CREATED_BY_ID" VARCHAR2(100), 
  "UPDATED_ON" DATE, 
  "UPDATED_BY_ID" VARCHAR2(100), 
  "PROCESS_STATUS" VARCHAR2(1) DEFAULT 'N' NOT NULL ENABLE, 
  "ASSET_TECH_ID" NUMBER, 
  "WORKFLOW_FLAG" VARCHAR2(1) DEFAULT 'N' NOT NULL ENABLE, 
  CONSTRAINT "RM_ASSET_HISTORY_PK" PRIMARY KEY ("****_ID") USING INDEX PCTFREE

现在有几个疑问想请教大家:

  • 这个问题会不会是因为特定表缺少快照或者元数据导致的?
  • 当表存在PRIMARY KEY或者NOT NULL这类约束时,Debezium LogMiner的表现会有差异吗?
  • 这种情况有没有什么恢复或者重新初始化DB history的建议?

非常感谢大家的帮助!

内容来源于stack exchange

火山引擎 最新活动