Clickhouse-backup恢复后数据库为空的问题排查求助

环境与备份信息 #

• ClickHouse集群配置：1分片，5副本，部署在Kubernetes环境
• 远程备份列表（执行clickhouse-backup list输出）：

  clickhouse-backup list
  2022-09-05T14-28-31   73.71KiB    05/09/2022 14:29:03   remote      tar, regular
  2022-09-05T23-47-29   541.83MiB   05/09/2022 23:48:46   remote      tar, regular
  2022-09-06T20-43-43   52.16MiB    06/09/2022 20:44:15   remote      tar, regular
  

  注：第一个73.71KiB的备份为权限配置错误导致的仅元数据备份，本次恢复目标为2022-09-05T23-47-29的完整备份。

执行的恢复操作 #

• 所有5个副本节点执行：

  clickhouse-backup restore_remote --rm --schema 2022-09-05T23-47-29
  clickhouse-backup delete local 2022-09-05T23-47-29
  

• 第一个副本节点额外执行：

  clickhouse-backup restore_remote --rm 2022-09-05T23-47-29
  clickhouse-backup delete local 2022-09-05T23-47-29
  

异常现象 #

1. 恢复过程中大量触发如下警告日志：

   2022/09/07 20:34:20.890442  info CREATE TABLE foo.bar (`project` String, `taskId` String, `addedAt` Nullable(DateTime('Europe/Copenhagen')), `metadata` String, `userId` String, `domain` String) ENGINE = ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/foo/bar', '{replica}') PARTITION BY project ORDER BY (domain, project) SETTINGS index_granularity = 8192
   2022/09/07 20:34:21.016010  warn can't create table 'foo.bar': code: 253, message: Replica /clickhouse/clickhouse/app-staging/tables/0/foo/bar/replicas/chi-clickhouse-app-staging-0-2 already exists, will try again backup=2022-09-05T23-47-29 operation=restore
   

2. 执行clickhouse-backup tables命令后，显示多个大小为0B的表，这些表在恢复前可正常使用。

内容的提问来源于stack exchange，提问作者Alexander K