HAProxy http-check无法检测Go运行时以外的mysqld-exporter指标

后端配置 #

backend galera-nodes
  mode tcp
  option srvtcpka
  option httpchk GET /metrics
  #http-check expect string go_threads <- 这是列表末尾的指标，可正常工作
  # http-check expect string mysql_engine_innodb_read_views_open_inside_innodb < 这是首个非Go运行时指标，无法工作      
  http-check expect string mysql_global_variables_wsrep_on\ 1 # 我希望该指标能正常检测，但未能成功
  balance first

  default-server init-addr none check resolvers mydns
  server node-0 mariadb-0.mariadb.test-mariadb.svc.cluster.local:3306 check port 9104
  server node-1 mariadb-1.mariadb.test-mariadb.svc.cluster.local:3306 check port 9104 backup
  server node-2 mariadb-2.mariadb.test-mariadb.svc.cluster.local:3306 check port 9104 backup

问题现象 #

• 可正常检测列表顶部的Go运行时相关指标，例如：

# HELP go_gc_cycles_automatic_gc_cycles_total Count of completed GC cycles generated by the Go runtime.
# TYPE go_gc_cycles_automatic_gc_cycles_total counter
go_gc_cycles_automatic_gc_cycles_total 148211
# HELP go_gc_cycles_forced_gc_cycles_total Count of completed GC cycles forced by the application.
# TYPE go_gc_cycles_forced_gc_cycles_total counter
go_gc_cycles_forced_gc_cycles_total 0

• 无法检测go_threads和mysql_exporter_collector_duration_seconds之后的所有指标，例如mysql_exporter_last_scrape_error：

# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 10
# HELP mysql_engine_innodb_read_views_open_inside_innodb Read views open inside InnoDB.
# TYPE mysql_engine_innodb_read_views_open_inside_innodb gauge
mysql_engine_innodb_read_views_open_inside_innodb 0
# HELP mysql_exporter_collector_duration_seconds Collector time duration.
# TYPE mysql_exporter_collector_duration_seconds gauge
mysql_exporter_collector_duration_seconds{collector="collect.engine_innodb_status"} 0.000647209
mysql_exporter_collector_duration_seconds{collector="collect.global_status"} 0.006044561
mysql_exporter_collector_duration_seconds{collector="collect.global_variables"} 0.010484458
mysql_exporter_collector_duration_seconds{collector="collect.info_schema.innodb_cmp"} 0.000210445
mysql_exporter_collector_duration_seconds{collector="collect.info_schema.innodb_cmpmem"} 0.000821883
mysql_exporter_collector_duration_seconds{collector="collect.info_schema.query_response_time"} 0.010943727
mysql_exporter_collector_duration_seconds{collector="collect.slave_status"} 0.000288631
mysql_exporter_collector_duration_seconds{collector="connection"} 0.00164831
# HELP mysql_exporter_last_scrape_error Whether the last scrape of metrics from MySQL resulted in an error (1 for error, 0 for success).
# TYPE mysql_exporter_last_scrape_error gauge
mysql_exporter_last_scrape_error 0 #### <--- 该指标无法被访问 

排查细节 #

检查指标输出的字符，未发现损坏或特殊字符：

# HELP go_threads Number of OS threads created.^M$
# TYPE go_threads gauge^M$
go_threads 10^M$
# HELP mysql_engine_innodb_read_views_open_inside_innodb Read views open inside InnoDB.^M$
# TYPE mysql_engine_innodb_read_views_open_inside_innodb gauge^M$
mysql_engine_innodb_read_views_open_inside_innodb 0^M$
# HELP mysql_exporter_collector_duration_seconds Collector time duration.^M$
# TYPE mysql_exporter_collector_duration_seconds gauge^M$
mysql_exporter_collector_duration_seconds{collector="collect.engine_innodb_status"} 0.000566157^M$
mysql_exporter_collector_duration_seconds{collector="collect.global_status"} 0.005119717^M$
mysql_exporter_collector_duration_seconds{collector="collect.global_variables"} 0.010971451^M$
mysql_exporter_collector_duration_seconds{collector="collect.info_schema.innodb_cmp"} 0.000200721^M$
mysql_exporter_collector_duration_seconds{collector="collect.info_schema.innodb_cmpmem"} 0.000738683^M$
mysql_exporter_collector_duration_seconds{collector="collect.info_schema.query_response_time"} 0.011611622^M$
mysql_exporter_collector_duration_seconds{collector="collect.slave_status"} 0.011444507^M$
mysql_exporter_collector_duration_seconds{collector="connection"} 0.001675709^M$
# HELP mysql_exporter_last_scrape_error Whether the last scrape of metrics from MySQL resulted in an error (1 for error, 0 for success).^M$
# TYPE mysql_exporter_last_scrape_error gauge^M$
mysql_exporter_last_scrape_error 0^M$
# HELP mysql_exporter_scrapes_total Total number of times MySQL was scraped for metrics.^M$
# TYPE mysql_exporter_scrapes_total counter^M$
mysql_exporter_scrapes_total 92990^M$

已尝试操作 #

• 检测不同位置的指标
• 使用string、rstring进行搜索匹配
• 手动验证指标可正常访问
• 通过curl和telnet访问/metrics端点均正常

版本信息 #

• HAProxy版本：HAProxy version 2.8.1-a90123a, released 2023/07/03
• mysqld-exporter版本：0.14.0-debian-11-r137

疑问与期望 #

是否是HAProxy的缓冲区限制导致的问题？有没有人遇到过类似情况？

期望HAProxy能够正常获取并检测目标指标。

内容的提问来源于stack exchange，提问作者wejdross