Vector采集主机指标写入CnosDB时触发数据库崩溃问题排查

我尝试通过Vector采集主机指标并写入CnosDB以构建可视化监控系统，配置如下：

data_dir = "/var/lib/vector"

[sources.hostmetrics001]
type = "host_metrics"

[transforms.tf001hostmetrics001]
type = "filter"
inputs = ["hostmetrics001"]
condition = '.name != "filesystem_used_ratio"'

[transforms.tfhostmetrics001]
type = "remap"
inputs = ["tf001hostmetrics001"]
source = """
.tags._tenant = "cnosdb"
.tags._database = "hostmetrics"
.tags._user = "root"
.tags._password = ""
"""
[sinks.skhostmetrics001]
type = "vector"
inputs = ["tfhostmetrics001"]
address = "127.0.0.1:12006"

执行启动命令：

sudo vector --config-toml /etc/vector/vector.toml 

Vector日志显示请求重试告警：

2023-09-20T02:53:47.849792Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=info,rdkafka=info,buffers=info,lapin=info,kube=info"
2023-09-20T02:53:47.862010Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2023-09-20T02:53:47.868310Z  INFO source{component_kind="source" component_id=hostmetrics001 component_type=host_metrics component_name=hostmetrics001}: vector::sources::host_metrics: PROCFS_ROOT is unset. Using default '/proc' for procfs root.
2023-09-20T02:53:47.868379Z  INFO source{component_kind="source" component_id=hostmetrics001 component_type=host_metrics component_name=hostmetrics001}: vector::sources::host_metrics: SYSFS_ROOT is unset. Using default '/sys' for sysfs root.
2023-09-20T02:53:47.885149Z  INFO vector::topology::running: Running healthchecks.
2023-09-20T02:53:47.885348Z  INFO vector: Vector has started. debug="false" version="0.31.0" arch="x86_64" revision="0f13b22 2023-07-06 13:52:34.591204470"
2023-09-20T02:53:47.885410Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2023-09-20T02:53:47.927543Z  INFO vector::topology::builder: Healthcheck passed.
2023-09-20T02:53:49.046900Z  WARN sink{component_kind="sink" component_id=skhostmetrics001 component_type=vector component_name=skhostmetrics001}:request{request_id=1}: vector::sinks::util::retries: Retrying after error. error=Request failed: status: Cancelled, message: "h2 protocol error: http2 error: stream error received: stream no longer needed", details: [], metadata: MetadataMap { headers: {} } internal_log_rate_limit=true
2023-09-20T02:53:50.090529Z  WARN sink{component_kind="sink" component_id=skhostmetrics001 component_type=vector component_name=skhostmetrics001}:request{request_id=1}: vector::sinks::util::retries: Internal log [Retrying after error.] is being suppressed to avoid flooding.

同时CnosDB发生崩溃，日志如下：

The application panicked (crashed).
Message:  begin <= end (34 <= 10) when slicing `mountpoint=/sys/fs/cgroup/net_cls,net_prio,collector=filesystem,host=cicd_ujv23.100020,filesystem=cgroup,device=cgroup,metric_type=gauge value=0 1695178427886213462
host.filesystem_total_bytes,device=cgroup,collector=filesystem,mountpoint=/sys/fs/cgroup/ne`[...]
Location: common/protocol_parser/src/lib.rs:235

请问导致CnosDB崩溃的原因是什么？

原因分析与解决方案 #

核心原因 #

CnosDB崩溃是因为协议解析模块处理包含特殊字符的标签值时触发了致命错误：

• 从崩溃日志的begin <= end (34 <= 10)错误可以看出，CnosDB在解析mountpoint=/sys/fs/cgroup/net_cls,net_prio这类包含逗号的路径标签时，错误地将逗号作为字段分隔符进行切割，导致字符串切片的起始索引大于结束索引，触发了Rust语言的panic（Rust不允许这种非法切片操作）。
• 你使用Vector的vector类型sink发送数据，Vector原生协议的格式在CnosDB的解析逻辑中存在兼容性问题，尤其是对包含逗号的标签值处理存在bug。

解决方法 #

• 方案1：改用CnosDB兼容的InfluxDB协议发送数据
  CnosDB原生兼容InfluxDB Line Protocol，修改Vector的sink配置为influxdb类型：

  [sinks.skhostmetrics001]
  type = "influxdb"
  inputs = ["tfhostmetrics001"]
  endpoint = "http://127.0.0.1:8902" # CnosDB默认InfluxDB协议端口
  database = "hostmetrics"
  username = "root"
  password = ""
  

• 方案2：在Vector中处理特殊字符
  在remap转换步骤中，将标签值中的逗号替换为其他合法字符（如下划线）：

  [transforms.tfhostmetrics001]
  type = "remap"
  inputs = ["tf001hostmetrics001"]
  source = """
  .tags._tenant = "cnosdb"
  .tags._database = "hostmetrics"
  .tags._user = "root"
  .tags._password = ""
  # 替换mountpoint中的逗号
  if exists(.tags.mountpoint) {
    .tags.mountpoint = replace(.tags.mountpoint, ",", "_")
  }
  """
  

• 方案3：升级CnosDB到最新版本
  该字符串切片panic属于CnosDB的已知bug，后续版本大概率已经修复，升级到最新稳定版即可解决。

内容的提问来源于stack exchange，提问作者Baker X