MySQL2ByteHouse_CDW 实时整库同步常见问题 #

表结构中或解决方案配置时带切分键，任务 oom #

• 问题现象：
  MySQL2ByteHouse_CDW 实时整库同步方案执行全量同步任务时，源端表数据量比较大时，日志提示 OutOfMemoryError: Java heap space，详细错误日志如下：

  Caused by: java.lang.OutOfMemoryError: Java heap space
          at java.math.BigInteger.<init>(BigInteger.java:1123) ~[?:1.8.0_181]
          at java.math.BigInteger.valueOf(BigInteger.java:1107) ~[?:1.8.0_181]
          at com.bytedance.dts.batch.jdbc.split.QuickSplitOneShardCallable.calculateRanges(QuickSplitOneShardCallable.java:47) ~[?:?]
          at com.bytedance.dts.batch.jdbc.split.SplitOneShardCallable.call(SplitOneShardCallable.java:206) ~[?:?]
          at com.bytedance.dts.batch.jdbc.split.SplitOneShardCallable.call(SplitOneShardCallable.java:45) ~[?:?]
          at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
          at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
  

• 解决方案：

  • 方案一：解决方案配置时，去掉切分键配置
  • 方案二：在解决方案配置 > 资源组高级配置 > 离线全量同步高级参数配置中，配置以下高级参数：job.reader.shard_split_mode=nosplit、job.common.is_use_batch_mode=false
    

批量写入分区限制数问题 #

• 问题现象：
  MySQL2ByteHouse_CDW 实时整库同步中离线同步任务执行失败，错误关键字提示：errbook_Gateway_InsertToWorker_WaitFail_IEToo many partitions for single INSERT block (more than 100)，详细错误日志如下：

  2024-11-07 14:17:18
  java.io.IOException: Couldn't write data - {"byteSize":8,"rawData":1800003},{"byteSize":0,"rawData":""},{"byteSize":24,"rawData":"m_O9qaP1s2uci0r_O9qaP1s2"},{"byteSize":93,"rawData":"http://1252524126.vod2.myqcloud.com/522ff1e0vodcq1252524126/5f394e04387702291367856795/f0.aac"},{"byteSize":0,"rawData":""},{"byteSize":32,"rawData":"cb9961e1403552fe3fa0f3df6379f243"},{"byteSize":8,"rawData":1},{"byteSize":8,"rawData":"30749.92"},{"byteSize":8,"rawData":0},{"byteSize":8,"rawData":0},{"byteSize":0},{"byteSize":8,"rawData":0},{"byteSize":0,"subType":"DATETIME"},{"byteSize":8,"rawData":1636772979000,"subType":"DATETIME"},{"byteSize":8,"rawData":1652747646000,"subType":"DATETIME"}
   at com.bytedance.bitsail.core.bytedance.flink.bridge.writer.delegate.DelegateFlinkWriter.processElement(DelegateFlinkWriter.java:302)
   at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:191)
   at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:188)
   at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:163)
   at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
   at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:478)
   at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:191)
   at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:930)
   at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:876)
   at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:887)
   at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:860)
   at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:1036)
   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:807)
   at java.lang.Thread.run(Thread.java:748)
  Caused by: java.lang.IllegalStateException: com.bytedance.bitsail.common.BitSailException: Code:[Common-05], Describe:[Internal error found in BitSail. This is usually a bug.] - com.bytedance.bytehouse.exception.ByteHouseSQLException: errbook_Gateway_InsertToWorker_WaitFail_IEToo many partitions for single INSERT block (more than 100). The limit is controlled by 'max_partitions_per_insert_block' setting. Large number of partitions is a common misconception. It will lead to severe negative performance impact, including slow server startup, slow INSERT queries and slow SELECT queries. Recommended total number of partitions for a table is under 1000..10000. Please note, that partitioning is not intended to speed up SELECT queries (ORDER BY key is sufficient to make range queries fast). Partitions are intended for data manipulation (DROP PARTITION, etc). SQLSTATE: HY000. Stack trace:
  

• 解决方案：
  该错误提示根因是因为目前离线任务默认批量写入的分区数限制为100，如果分区数超过100，则会报错。您可在解决方案离线全量同步中添加集成高级参数： job.writer.bh_connection_properties={"max_partitions_per_insert_block":"10000"}

缺失唯一键：errbook_Gateway_InsertToWorker_WaitError_IENo such column delete_flag in table #

• 问题现象：
  MySQL2ByteHouse_CDW 实时整库同步中离线同步任务执行失败，错误关键字提示：errbook_Gateway_InsertToWorker_WaitError_IENo such column delete_flag in table，详细错误日志如下：

  2024-11-07 12:17:43,176 ERROR org.apache.flink.runtime.executiongraph.ExecutionGraph       flink-akka.actor.default-dispatcher-17                       [] - bytehouse_writer(1/4) - execution #0 (da959709c09229c65fa17737766cec9a) switched from RUNNING to FAILED on s-1854376938343854082-taskmanager-1-2 @ 33.1.192.126 (dataPort=39739) java.io.IOException: Couldn't write data - {"byteSize":8,"rawData":50000},{"byteSize":15,"rawData":"appyMlpdpb01900"},{"byteSize":36,"rawData":"099aad74-3131-48e5-926e-44b3185ac6db"},{"byteSize":102,"rawData":"http://wechatapppro-1252524126.file.myqcloud.com/appyMlpdpb01900/image/ueditor/93591500_1540728476.jpg"},{"byteSize":0,"rawData":""},{"byteSize":32,"rawData":"8468cafad05789f05bbc2911f2c925dc"},{"byteSize":8,"rawData":1},{"byteSize":5,"rawData":"14.99"},{"byteSize":8,"rawData":0},{"byteSize":8,"rawData":0},{"byteSize":0},{"byteSize":8,"rawData":0},{"byteSize":8,"rawData":1643212527000,"subType":"DATETIME"},{"byteSize":8,"rawData":1579282487000,"subType":"DATETIME"},{"byteSize":8,"rawData":1579282564000,"subType":"DATETIME"}
   at com.bytedance.bitsail.core.bytedance.flink.bridge.writer.delegate.DelegateFlinkWriter.processElement(DelegateFlinkWriter.java:302)
   at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:191)
   at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:188)
   at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:163)
   at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
   at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:478)
   at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:191)
   at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:930)
   at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:876)
   at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:887)
   at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:860)
   at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:1036)
   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:807)
   at java.lang.Thread.run(Thread.java:748)
  Caused by: java.lang.IllegalStateException: com.bytedance.bitsail.common.BitSailException: Code:[Common-05], Describe:[Internal error found in BitSail. This is usually a bug.] - java.lang.RuntimeException: com.bytedance.bytehouse.exception.ByteHouseSQLException: errbook_Gateway_InsertToWorker_WaitError_IENo such column _delete_flag_ in table `2100132197.db_material_center`.t_material_extend (3cadfe8c-5ae1-4b59-ba00-28936047fa02) SQLSTATE: 42703. Stack trace:
  

• 解决方案：
  实时整库同步时以离线+实时的方式，数据写入方式都是以 Upsert 方式写入，因此已创建的目标 ByteHouse CDW 表必须要设置UNIQUE KEY，否则会导致写入失败。表设置操作详见唯一键表。

CDW 没有默认资源组： BytehouseBaseResp(message=null, data=null, error=Error(message=no virtual warehouse was selected), meta=null) #

• 问题现象：
  MySQL2ByteHouse_CDW 实时整库同步中自动建表失败，错误关键字提示：Failed to create table, remote response: BytehouseBaseResp(message=null, data=null, error=Error(message=no virtual warehouse was selected), meta=null)，详细错误日志如下：

  com.bytedance.dataplatform.dts.solution.exception.SolutionException: Create table error., info: Failed to create table, remote response: BytehouseBaseResp(message=null, data=null, error=Error(message=no virtual warehouse was selected), meta=null)
  ErrorStack
  com.bytedance.dataplatform.dts.manager.ByteHouseCdwManager.createTable(ByteHouseCdwManager.java:418) com.bytedance.dataplatform.dts.manager.ByteHouseCdwManager$$FastClassBySpringCGLIB$$552b7ae2.invoke(<generated>) org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
  

• 解决方案：
  ByteHouse CDW 租户管理控制台中，没有设置默认计算组参数，需要手动设置，设置方法详见设置计算组。
  

创建 DataSail 一次性全量批任务时失败，多次尝试后出现重名任务报错、启动时找不到任务等问题，影响数据导入测试。 #

• 问题现象：
  DataSail 一次性全量批任务创建失败，报错“任务名称唯一性检查：当前项目下已存在同名任务”，但实际项目中不存在该同名任务。详细错误日志如下：

  java.lang.RuntimeException: Create task failed: call StudioCreateTask error, code: 11 due to 任务: ai_usage_record_b_mysql2BH_10004790_c686b81j52df [任务名称唯一性检查]:当前项目下已存在同名任务:ai_usage_record_b_mysql2BH_10004790_c686b81j52df;
  ErrorStack
  com.bytedance.dataplatform.dts.solution.manager.AbstractApiManager.handleError(AbstractApiManager.java:176) com.bytedance.dataplatform.dts.solution.manager.AbstractApiManager.handleDataLeapError(AbstractApiManager.java:180) com.bytedance.dataplatform.dts.solution.manager.studio.StudioApiManager.handleDataLeapError(StudioApiManager.java:935) com.bytedance.dataplatform.dts.solution.manager.Ab
  

• 解决方案：
  检查一下该任务是否配置了相关的复查策略。若任务配置了复查策略会导致创建失败，需要先手动关闭复查策略，待任务创建成功后再开启。管理复查策略操作请参见配置发布复查策略。