VictoriaMetrics集群数据异常：Grafana大范围无数据、放大可见问题排查

我们尝试向VictoriaMetrics集群回填6个月的数据，集群部署包含2个vmstorage节点、1个vminsert和1个vmselect实例。回填过程起初正常，但Grafana中某一特定日期后的数据消失，导入脚本仍正常运行且vminsert与vmstorage日志无报错，仅时间范围末端有数据显示。

Grafana中存在大量数据缺失，但有时放大时间范围后会显示部分原本不出现的数据。核对vmstorage日志发现，两个节点的分区不一致，其中一个节点缺少最新的分区：

存储节点1日志 #

2022-12-02T00:21:22.521Z    info    VictoriaMetrics/lib/storage/partition.go:200    creating a partition "2022_06" with smallPartsPath="/storage/data/small/2022_06", bigPartsPath="/storage/data/big/2022_06"
2022-12-02T00:21:24.259Z    info    VictoriaMetrics/lib/storage/partition.go:216    partition "2022_06" has been created
2022-12-02T13:56:23.722Z    info    VictoriaMetrics/lib/storage/partition.go:200    creating a partition "2022_07" with smallPartsPath="/storage/data/small/2022_07", bigPartsPath="/storage/data/big/2022_07"
2022-12-02T13:56:26.533Z    info    VictoriaMetrics/lib/storage/partition.go:216    partition "2022_07" has been created
2022-12-03T01:24:45.721Z    info    VictoriaMetrics/lib/storage/partition.go:200    creating a partition "2022_08" with smallPartsPath="/storage/data/small/2022_08", bigPartsPath="/storage/data/big/2022_08"
2022-12-03T01:24:46.900Z    info    VictoriaMetrics/lib/storage/partition.go:216    partition "2022_08" has been created
2022-12-03T17:57:02.525Z    info    VictoriaMetrics/lib/storage/partition.go:200    creating a partition "2022_09" with smallPartsPath="/storage/data/small/2022_09", bigPartsPath="/storage/data/big/2022_09"
2022-12-03T17:57:03.713Z    info    VictoriaMetrics/lib/storage/partition.go:216    partition "2022_09" has been created
2022-12-04T07:08:51.722Z    info    VictoriaMetrics/lib/storage/partition.go:1305   merged 18251 rows across 18251 blocks in 37.328 seconds at 488 rows/sec to "/storage/data/small/2022_09/18251_18251_20220928180000.000_20220928180000.000_172D5A35BF48B011"; sizeBytes: 250837
2022-12-04T08:41:28.530Z    info    VictoriaMetrics/lib/storage/partition.go:200    creating a partition "2022_10" with smallPartsPath="/storage/data/small/2022_10", bigPartsPath="/storage/data/big/2022_10"
2022-12-04T08:41:31.022Z    info    VictoriaMetrics/lib/storage/partition.go:216    partition "2022_10" has been created
2022-12-04T21:22:29.569Z    info    VictoriaMetrics/lib/mergeset/table.go:1027  merged 24870725 items across 28659 blocks in 37.205 seconds at 668480 items/sec to "/storage/indexdb/172CAC7059FB5EB3/24860954_28638_172CAC71E7BD8BD0"; sizeBytes: 331467849
2022-12-04T22:37:55.719Z    info    VictoriaMetrics/lib/storage/partition.go:200    creating a partition "2022_11" with smallPartsPath="/storage/data/small/2022_11", bigPartsPath="/storage/data/big/2022_11"
2022-12-04T22:37:56.406Z    info    VictoriaMetrics/lib/storage/partition.go:216    partition "2022_11" has been created
2022-12-05T00:39:45.097Z    info    VictoriaMetrics/lib/storage/partition.go:1305   merged 199154 rows across 199154 blocks in 30.426 seconds at 6545 rows/sec to "/storage/data/small/2022_11/199154_113352_20221104180000.000_20221106180000.000_172DB81E235772F4"; sizeBytes: 1909359
2022-12-05T03:29:28.254Z    info    VictoriaMetrics/lib/storage/partition.go:1305   merged 211814 rows across 211814 blocks in 30.355 seconds at 6977 rows/sec to "/storage/data/small/2022_11/211814_114004_20221108180000.000_20221110180000.000_172DB81E235773BF"; sizeBytes: 1891824
2022-12-05T16:34:53.329Z    info    VictoriaMetrics/lib/storage/partition.go:200    creating a partition "2022_12" with smallPartsPath="/storage/data/small/2022_12", bigPartsPath="/storage/data/big/2022_12"
2022-12-05T16:34:54.324Z    info    VictoriaMetrics/lib/storage/partition.go:216    partition "2022_12" has been created

存储节点2日志 #

2022-12-02T00:21:22.523Z    info    VictoriaMetrics/lib/storage/partition.go:200    creating a partition "2022_06" with smallPartsPath="/storage/data/small/2022_06", bigPartsPath="/storage/data/big/2022_06"
2022-12-02T00:21:23.985Z    info    VictoriaMetrics/lib/storage/partition.go:216    partition "2022_06" has been created
2022-12-02T13:56:23.727Z    info    VictoriaMetrics/lib/storage/partition.go:200    creating a partition "2022_07" with smallPartsPath="/storage/data/small/2022_07", bigPartsPath="/storage/data/big/2022_07"
2022-12-02T13:56:26.533Z    info    VictoriaMetrics/lib/storage/partition.go:216    partition "2022_07" has been created
2022-12-03T01:24:45.724Z    info    VictoriaMetrics/lib/storage/partition.go:200    creating a partition "2022_08" with smallPartsPath="/storage/data/small/2022_08", bigPartsPath="/storage/data/big/2022_08"
2022-12-03T01:24:46.900Z    info    VictoriaMetrics/lib/storage/partition.go:216    partition "2022_08" has been created
2022-12-03T17:57:02.517Z    info    VictoriaMetrics/lib/storage/partition.go:200    creating a partition "2022_09" with smallPartsPath="/storage/data/small/2022_09", bigPartsPath="/storage/data/big/2022_09"
2022-12-03T17:57:03.713Z    info    VictoriaMetrics/lib/storage/partition.go:216    partition "2022_09" has been created
2022-12-04T07:08:23.316Z    info    VictoriaMetrics/lib/storage/partition.go:1305   merged 23345 rows across 23345 blocks in 33.611 seconds at 694 rows/sec to "/storage/data/small/2022_09/23345_23345_20220928180000.000_20220928180000.000_172D5A35BF445C06"; sizeBytes: 338817
2022-12-04T08:41:28.524Z    info    VictoriaMetrics/lib/storage/partition.go:200    creating a partition "2022_10" with smallPartsPath="/storage/data/small/2022_10", bigPartsPath="/storage/data/big/2022_10"
2022-12-04T08:41:31.022Z    info    VictoriaMetrics/lib/storage/partition.go:216    partition "2022_10" has been created
2022-12-04T22:37:55.725Z    info    VictoriaMetrics/lib/storage/partition.go:200    creating a partition "2022_11" with smallPartsPath="/storage/data/small/2022_11", bigPartsPath="/storage/data/big/2022_11"
2022-12-04T22:37:56.406Z    info    VictoriaMetrics/lib/storage/partition.go:216    partition "2022_11" has been created

1. 当前问题的根源是什么？ #

根源就是两个vmstorage节点的数据分区不一致：存储节点1已经创建了2022_12分区，存储节点2却没有这个分区，而且部分分区的数据合并状态也存在差异。

回填数据时，vminsert是按一致性哈希规则把数据分发到不同的vmstorage节点的，如果某个节点没成功创建对应时间的分区（可能是磁盘IO卡顿、节点资源不足、分区创建触发逻辑未执行等原因），该节点就无法存储对应时间段的数据。当vmselect查询时，会从所有配置的vmstorage节点拉取数据并聚合，缺分区的节点拿不出对应时间段的数据，最终导致Grafana中显示数据缺失。

2. 为何放大时间范围后有时能看到原本缺失的数据？ #

这和VictoriaMetrics的查询聚合逻辑有关：

• 查询大时间范围时，vmselect会聚合所有节点中存在的数据，如果缺失的时间段里，其中一个节点还存着部分数据，这部分数据就会被返回，所以看起来原本缺失的数据又出现了。
• 小时间范围查询时，可能刚好命中了缺分区节点负责的数据分片，导致无数据返回；而放大时间范围后，查询窗口覆盖到了有数据的分片，自然就能看到部分内容。

3. 是否因请求路由到不同节点导致数据显示不一致？ #

不是。vmselect的工作机制是向所有配置的vmstorage节点发起查询请求，然后将所有节点返回的结果聚合在一起，而不是把请求路由到单个节点。

数据显示异常的核心是节点间数据不完整：部分节点缺少对应时间的分区，聚合后的结果就会缺失部分数据；偶尔能看到数据是因为另一节点存有该时间段的部分数据，查询时被聚合进来了。

内容的提问来源于stack exchange，提问作者Sakibul Alam